I will try the test of switching out KEPCOs one at a time for the HP. Given that the passive RC filter doesn't filter out the excess, I am wondering if the KEPCO is somehow polluting the circuit ground? The measurement was made between the circuit side of R24 (see schematic) and a ground testpoint, so the passive R23/C15 pole should filter the noise above ~15 Hz.
This is very disappointing. Even with KEPCO linear supply with the improved HV driver circuit, the noise level is significantly higher than the 20kOhm R thermal noise.
What is special with the HP supplies? Can you replace KEPCOs with the HP supply, one by one to specify which one is making the noise bad?
On Friday evening we checked out a few more things, somewhat overlapping with previous tests. All tests done with PRMI on carrier lock (REFL11_I -> PRC, AS55_Q-> MICH):
unrelated note: Donatella the Workstation was ~3 minutes ahead of the FE machines (you can look at the C0:TIM-PACIFIC_STRING on many of the MEDM screens for a rough simulacrum). When the workstation time is so far off, DTT doesn't work right (has errors like test timed out, or other blah blah). I installed NTP on donatella and started the service per SL7 rules. Since we want to migrate all the workstations to Debian (following the party line), lets not futz with this too much.
gautam, 1 Mar 1600: In case I'm being dumb, I attach the screen grab comparing dark offset to the single bounce off PRM, to estimate the RAM contribution. The other signals are there just to show that the ITMs are sufficiently misaligned. The PRCL PDH fringe is usually ~12000 cts in REFL11, ~5000cts in REFL55, and so the RAM offset is <0.1% of the horn-to-horn PDH fringe.
P.S. I know generally PNGs in the elog are frowned upon. But with so many points, the vector PDF export by NDS (i) is several megabytes in size and (ii) excruciatingly slow. I'm proposing a decimation filter for the export function of ndscope - but until then, I claim plotting with "rasterized=True" and saving to PDF and exporting to PNG are equivalent, since both yield a rasterized graphic.
I looked into this a bit more and crossed off some of the points Rana listed. In order to use REFL 55 as a sensor, I had to fix the frequent saturations seen in the MICH signals, at the nominal (flat) whitening gain of +18 dB. The light level on the REFL55 photodiode (13 mW), its transimpedance (400 ohm), and this +18dB (~ x8) gain, cannot explain signal saturation (0.7A/W * 400 V/A * 8 ~ 2.2kV/W, and the PRCL PDH fringe should be ~1 MW/m, so the PDH fringe across the 4nm linewidth of the PRC should only be a couple of volts). Could be some weird effect of the quad LT1125. Anyway, the fix that has worked in the past, and also this time, is detailed here. Note that the anomalously high noise of the REFL55_Q channel in particular remains a problem. After taking care of that, I did the following:
Rana also suggested checking if the digital demod phase that senses MICH in REFL55_Q changes from free-swinging Michelson (PRM misaligned), to PRMI aligned - we can quantify any macroscopic length mismatch in the PRC length using this measurement. I couldn't see any MICH signal in REFL55_Q with the PRM misaligned and the Michelson fringing. Could be that +18dB is insufficient whitening gain, but I ran out of time this afternoon, so I'll check later. But not sure if the double attenuation by the PRM makes this impossible.
The PRM violin filter seems very suboptimal - the gain peaking shows up in the MICH OLTF, presumably due to the MICH-->PRM LSC output matrix. I plot the one used for the BS in comparison in Attachment #1, seems much more reasonable. Why does the PRM need so many notches? Is this meant to cover some violin modes of PR2/PR3 as well? Do we really need that? Are the PR2/PR3 violin modes really so close in frequency to that for the 3" SOS? I suppose it could be since the suspension wire is thinner and the mass is lighter, and the two effects nearly cancel, but we don't actuate on PR2/PR3? According to the earlier elog in this thread, this particular filter wasn't deemed offensive and was left on.
Indeed, as shown in Attachment #2, I can realize a much healthier UGF for the MICH loop with just a single frequency notch (black reference trace) rather than using the existing "PRvio1,2" filter (FM2), (live red trace). The PR violins are eating so much phase at ~600 Hz.
We turned off many excessive violin mode bandstop filters in the LSC.
agreed, seems excessive. I always prefer bandstop over notch in case the eigenfrequency wanders, but the bandstop could be made to be just a few Hz wide.
There were multiple problems with the REFL55 demod board. I fixed them and re-installed the board. The TFs and noise measured on the bench now look more like what is expected from a noise model. The noise in-situ also looked good. After this work, my settings for the PRMI sideband lock don't work anymore so I probably have to tweak things a bit, will look into it tomorrow.
I finished testing the OSEMs. I put all the OSEMs back in the box. The OSEMS were divided into several bags. I put the OSEM box next to the south flow bench on the floor.
I have uploaded the OSEM catalog to the wiki. I will upload the LED spot images later.
Total 64 OSEMS, 31 long, 33 short.
Perfectly centered LED spots, ready for C&B OSEMS: 30, 12 long, 18 short.
Perfectly centered LED spots, need some work (missing pigtails, weird screws) OSEMS: 7, 5 long, 2 short.
Slightly off-centered (subjective) LED spots, ready for C&B OSEMS: 20, 7 long, 13 short.
Slightly off-centered (subjective) LED spots, need some work (missing pigtails, weird screws) OSEMS: 4 long
Defective OSEMS or LED spot way off-center: 3.
After this work, I measured that the orthogonality was poor. I confirmed on the bench that the PQW-2-90 was busted, pin 2 (0 degree output) showed a sensible signal half of the input, but pin 6 had far too small an output and the phase difference was more like 45 degrees and not 90 degrees. I can't find any spares of this part in the lab - however, we do have the equivalent part used in the aLIGO demodulator. Koji has kindly agreed to do the replacement (it requires a bit of jumper wiring action because the pin mapping between the two parts isn't exactly identical - in fact, the circuit schematic uses a transformer to do the splitting, but at some unknown point in time, the change to the minicircuits part was made. Anyway, until this is restored, I defer the PRMI sideband locking.
I see no evidence of anything radically different from my PSL table optical characterization in the IMC transmitted beam, see Attachment #1. The lines are just a quick indicator of what's what and no sophisticated peak fitting has been done yet (so the apparent offset between the transmission peaks and some of the vertical lines are just artefacts of my rough calibration I believe). The modulation depths recovered from this scan are in good agreement with what I report in the linked elog, ~0.19 for f1 and ~0.24 for f2. On the bright side, the ALS just worked and didn't require any electronics fudgery from me. So the mystery continues.
A new hybrid splitter (DQS-10-100) was installed. As the amplification of the final stage is sufficient for the input level of 3dBm, I have bypassed the input amplification (Attachment 1). One of the mixer was desoldered to check the power level. With a 1dB ATTN, the output of the last ERA-5 was +17.8dBm (Attachment 2). (The mixer was resoldered.)
With LO3dBm. RF0dBm, and delta_f = 30Hz, the output Vpp of 340mV and the phase difference is 88.93deg. (Attachment 3/4, the traces were averaged)
0 dBm ~ 0.63 Vpp. I guess there is ~4dB total loss (3dB from splitter and 1dB from total excess loss above theoretical from various components) between the SMA input and each RF input of the JMS-1-H mixer, which has an advertised conversion loss of ~6dB. So the RF input to each mixer, for 0dBm to the front panel SMA is ~-4dBm (=0.4 Vpp), and the I/F output is 0.34Vpp. So the conversion loss is only ~-1.5 dB? Seems really low? I assume the 0.34 Vpp is at the input to the preamp? If it's after the preamp, then the numbers still don't add up, because with the nominal 6dB conversion loss, the output. should be ~2Vpp? I will check it later.
Missed to note: The IF test was done at TP7 and TP6 using pomona clips i.e. brefore the preamp.
I don't have a good explanation why, but I too measured similar numbers to what Koji measured. The overall conversion gain for this board (including the +20dB gain from the daughter board) was measured to be ~5.3 V/V on the bench, and ~16000 cts/V in the CDS system (100Hz offset from the LO frequency). It would appear that the effective JMS-1-H conversion loss is <2dB. Seems fishy, but I can't find anything else obviously wrong with the circuit (e.g. a pre-amp for the RF signal that I missed, there is none).
I also attach the result of the measured noise at the outputs of the daughter board (i.e. what is digitized by the ADC), see Attachment #2. Apart from the usual forest of lines of unknown origin, there is still a significant excess above the voltage noise of the OP27, which is expected to be the dominant noise source in this configuration. Neverthelesss, considering that we have only 40dB of whitening gain, it is not expected that we see this noise directly in the digitized signal (above the ADC noise of ~1uV/rtHz). Note that the measured noise today, particularly for the Q channel, is significantly lower than before the changes were made.
Today I moved the c1bhd machine from the control room to a new test area set up behind (west of) the 1X6 rack. The test stand is pictured in Attachment 1. I assembled one of the new IO chassis and connected it to the host.
The chassis was then powered on and LED lights illuminated indicating that all the components have power. The assembled chassis is pictured in Attachment 2.
Following the procedure outlined T1900700, the system failed the very first test of the communications link between chassis and host, which is to check that all PCIe cards installed in both the host and the expansion chassis are detected. The Dolpin host adapter card is detected:
07:06.0 PCI bridge: Stargen Inc. Device 0102 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=0e, subordinate=0e, sec-latency=0
I/O behind bridge: 00002000-00002fff
Prefetchable memory behind bridge: 00000000c0200000-00000000c03fffff
Capabilities:  Power Management version 2
Capabilities:  MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities:  Express Downstream Port (Slot+), MSI 00
Capabilities:  Subsystem: Device 0000:0000
Kernel driver in use: pcieport
However the OSS PCIe adapter card linking the host to the IO chassis was not detected, nor were any of the cards in the expansion chassis. Gautam previously reported that the OSS card was not detected by the host (though it was not connected to the chassis then). Even now connected to the IO chassis, the card is still not detected. On the chassis-side OSS card, there is a red LED illuminated indicating "HOST CARD RESET" as pictured in Attachment 3. This may indicate a problem with the card on the host side. Still more debugging to be done.
Now that the REFL55 signal chain is capable of providing balanced, orthogonal readout of the two quadratures, I was able to recover the 1f SB resonant lock pretty easily. Ran sensing lines for ~5mins, still looks weird. But I didn't try to optimize anything / do other checks (e.g. actuate MICH using ITMs instead of BS) tonight, and I'm craving the Blueberry pie Rana left me. Will continue to do more systematic tests in the next days.
To my dismay, I found today that somebody had changed the oscillator frequencies for the sensing matrix infrastructure we have. The change happened 2 days and 2 hours ago (I write this at ~1230 on Saturday, 3/6), i.e. ~1030am on Thursday. According to the elog, this is when Anchal and Paco were working on the interferometer, but I can find no mention of these settings being changed. Not cool guys 😒 .
This was relatively easy to track down but I don't know what else may have been messed with. I don't understand how anything that was documented in the elog can lead to this weird doubling of the frequencies.
I have now restored the correct settings. The "sensing matrix" I posted last night is obviously useless.
I was also reminded today of the poor reliability of the LSC whitening electronics. Basically, there may be hidden saturations in all the channels that have a large DC value (e.g. the photodiode DC mon channels) due to the poor design of the cascaded gain stages. I was thinking about using the REFL DC channel to estimate the mode-matching into the PRC, but this has a couple of problems. Electronically, there may be some signal distortion due to the aforementioned problem. But in addition, optically, the estimation of mode-matching into the PRC by comparing REFL DC levels in single bounce off the PRM and the PRMI locked has the problem that the mode-matching is degenerate with the intra-cavity loss, which is of the same order as the mode mismatch (a percent or two I claiM). If Koji or someone else can implement the fix suggested by Hartmut for all the LSC whitening channels, that'd give us more faith in the signals. It may be less work than just replacing all the whitening filters with a better design (e.g. the aLIGO ISC whitening filter which implements the cascaded gain stages using single OP27s and more importantly has a 1 kohm series resistance with the input to the op amp (so the preceeding stage never has to drive > 10V/1kohms ~10mA of DC current) would presumably reduce distortion.
I understand this mst be frustrating for you. But we did not change these settings, knowingly atleast. We have documented all the things we did there. The only thing I can think of which could possibly change any of those channels are the scripts that we ran that are mentioned and the burt restore that we did on all channels (which wasn't really necessary). We promise to be more vigilant of changes that occur when we are present in future.
I realized I hadn't checked the PRM actuator as thoroughly as I had the others. I used the Oplev as a sensor to check the coil balancing, and I noticed that while all 4 coils show up with the expected 1/f^2 profile at the Oplev error point, the actuator gains seem imbalanced by a factor of ~5. The phase isn't flat because of some filters in the Oplev electronics I guess. The Oplev loops were disabled for the measurement, and the excitations were small enough that the beam stayed reasonably well centered on the QPD throughout. This seems very large to me - the values in the coil output filter gains lead me to expect more like a ~10% mismatch in the actuation strenghts, and similar tests on other optics in the past, e.g. ETMY, have yielded much more balanced results. I'm collecting some free-swinging PRM data now as an additional check. I verified that all the coils seem actuatable at least, by applying a 500 ct step at the offset of the coil output FM, and saw that the optic moved (it was such a test that revealed that MC1 had a busted actuator some time ago). If the eigenmode spectra look as expected, I think we can rule out broken magnets, but I suppose the magnets could still be not well matched in strength?
There are still many mysteries remaining - e.g. the MICH-->PRCL contribution still can't be nulled. But for now, I have the settings that keep the PRMI locked fairly robustly with REFL55I/Q or REFL165I/Q (I quadrature for PRCL, Q for MICH in both cases), see Attachment #1 and Attachment #2 respectively. For the 1f locking, the REFL55 digital demod phase was fine-tuned to minimize the frequency noise (generated by driving MC2) coupling to the Michelson readout (as the Michelson is supposed to be immune) - the coupling was measured to be ~60dB larger at the PRCL error point than MICH. There was still nearly unity coherence between my MC2 drive and the MICH error point demodulated at the drive frequency, but I was not able to null it any better than this. With the PRMI (ETMs misaligned) locked on the 1f signals, I measured Attachment #1 and used it to determine the demod phase that would best enable REFL165_I to be a PRCL sensor. Rana thinks that if there is some subtle effect in the marginally stable PRC, we would not see it unless we do a mode scan (time consuming to set up and execute). So I'm just going to push on with the PRFPMI locking - let's see if the clean arm mode forces a clean TEM00 mode to be resonant in the PRC, and if that can sort out the lack of orthogonality between MICH/PRCL in the 1f sensors (after all, we only care about the 3f signals in as much as they allow us to lock the interferometer). I'll try the PRMI with arms on ALS tomorrow eve.
I have no idea what to make of how the single frequency lines I am driving in MICH and PRCL show up in REFL11 and REFL33 - the signals are apparently completely degenerate (in optical quadrature). How this is possible even though the PRMI remains stably locked, POP22/POP110/AS110 report stable sideband buildup is not clear to me.
I finished ranking the OSEMS on the OSEM wiki page.
I also moved the OSEM data folder from /home/export/home to /users/public_html and created a soft link instead. I have done the same for the 40m_TIS folder that I uploaded there a while ago.
How were the statistics of them? i.e. # of Good OSEMs, # of OK OSEMs, etc...
Today I continued with assembly and testing of the new front-ends. The main progress is that the IO chassis is now communicating with the host, resolving the previously reported issue.
Unfortunately, though, it turns out one of the two (host-side) One Stop Systems PCIe cards sent from Hanford is bad. After some investigation, I ultimately resolved the problem by swapping in the second card, with no other changes. I'll try to procure another from Keith Thorne, along with some spares.
Also, two of the three switching power supplies sent from Livingston (250W Channel Well PSG400P-89) appear to be incompatible with the Trenton BPX6806 PCIe backplanes in these chassis. The power supply cable has 20 conductors and the connector on the board has 24. The third supply, a 650W Antec EA-650, does have the correct cable and is currently powering one of the IO chassis. I'll confirm this situation with Keith and see whether they have any more Antecs. If not, I think these supplies can still be bought (not obsolete).
I've gone through all the hardware we've received, checked against the procurement spreadsheet. There are still some missing items:
Once the PCIe communications link between host and IO chassis was working, I carried out the testing procedure outlined in T1900700. This performs a series checks to confirm basic operation/compatibility of the hardware and PCIe drivers. All of the cards installed in both the host and the expansion chassis are detected and appear correctly configured, according to T1900700. In the below tree, there is one ADC, one 16-ch DIO, one 32-ch DO, and one DolphinDX card:
+-05.0-[05-20]----00.0-[06-20]--+-00.0-[07-08]----00.0-----00.0 Contec Co., Ltd Device 86e2
| | +-03.0-[0e]--
| | +-04.0-[0f]--
| | +-06.0-[10-11]----00.0-----04.0 PLX Technology, Inc. PCI9056 32-bit 66MHz PCI <-> IOBus Bridge
| | +-07.0---
| | +-08.0---
| | +-0a.0---
| | \-0b.0---
| +-0a.0-[1e-1f]----00.0-[1f]----00.0 Contec Co., Ltd Device 8632
\-08.0-[21-2a]--+-00.0 Stargen Inc. Device 0101
Before I start building/testing RTCDS models, I'd like to move the new front ends to an isolated subnet. This is guaranteed to prevent any contention with the current system, or inadvertent changes to it.
Today I set up another of the Supermicro servers sent by Livingston in the 1X6 test stand area. The intention is for this machine to run a cloned, bootable image of the current fb1 system, allowing it to function as a bootserver and DAQ server for the FEs on the subnet.
However, this hard disk containing the fb1 image appears to be corrupted and will not boot. It seems to have been sitting disconnected in a box since ~2018, which is not a stable way to store data long term. I wasn't immediately able to recover the disk using fsck. I could spend some more time trying, but it might be most time-effective to just make a new clone of the fb1 system as it is now.
29 Good OSEMs, of which 1 is questionable (089) with PD voltage of 1.5V, 5 need some work (pigtailing, replace/remove/add screws). We have 4 pigtails. Schematics.
20 OK OSEMs (Slightly off-centered LED spot), of which 3 need some work (pigtailing, replace/remove/add screws).
13 Bad OSEMS (Way off-centered LED spot)
2 Defunct OSEMs
Good: 23 complete OSEMs + 5 good ones, which need soldering work (there are 4 pigtails and take one from a defunct OSEM).
OK: Use good 7 OSEMs for the sides. And keep some functional OSEMs as spares.
The interferometer can nearly be locked again. I was unable to fully hand off control from ALS-->RF, I suspect I may be using the wrong sign on the AO path (or some such other sub-optimal CM board settings). I'll hook up the SR785 and take some TFs tomorrow, that should give more insight into what's what. With the arms held off resonance, the PRMI acquires lock nearly instantly (REFL165 I for PRCL, REFL165 Q for MICH), and can stay locked nearly indefinitely, which is what I need so I can get the RF lock going. However the sensing matrix (for vertex DoFs, arms held off resonance) still makes no sense to me. The MICH loop has ~50 Hz UGF and the PRCL loop ~150 Hz. I think the MICH loop shape can be optimized a little for better low frequency suppression, but this isn't the show-stopper at the moment. For record-keeping, the ALS performance was excellent and other subsystems were nominal tonight.
The procedure is that the optic is kicked to excite it, and allowed to ring down for ~1ksec, with damping turned off. The procedure is repeated 15 times for some averaging.
Attachment #1 - sensor spectra from yesterday.
Attachment #2 - peaks using the naive diagonalization matrix from yesterday.
Attachment #3 - Data from ~1 year ago.
The y-axis in all plots is labelled as "cts/rtHz" but these are the DQed channels, which come after a "cts2um" CDS filter - so if that filter is accurate, them the y-axes may be read as um/rtHz.
I wonder if the September 2020 earthquake somehow damaged the PRM suspension, as this experiment would suggest that the problem is not only with the actuation. The data was gathered with the neutral position of the PRM (between kicks) being well aligned for PRMI, and the DC values of all the shadow sensors in this position is close to half-light (~1V, except for side which was more like 4V). Hard to say what exactly is happening since only the PIT DoF has the weird asymmetric peak shape instead of the expected Lorentzian - I would have thought that a damaged wire or broken magnet would affect all 4 DoFs but the F.C. spring experience on ETMY showed that anything is possible.
As I am sitting in the control room, the PRM suspension watchdog tripped again. This time, there is clearly no seismic activity. Yet, the BS suspension also shows a slight disturbance at the same time as the PRM. ITMY shows no perturbation though. My best hypothesis here is that the problem is electrical. In Attachment #1, you can see that all of the Sensors go to -6000 cts (whut?) for ~30 seconds. Zooming in to that segment in Attachment #2, it would appear that the light detected by the LED changed dramatically (went dark?) on all 5 coils. The 4 face coils have the same time constant but the side has a different one, but in any case, this level of light change in half a second is clearly not physical. Then the watchdog trips because this huge apparent motion elicits a kick from the damping loops.
The plots I attach are for the DQed sensor channels, so there is some digital filtering involved. But I confirmed that the signal doesn't go negative if I disable the input to the filter module. So it would seem that the voltage input to the ADC really chanegd polarity, seems unphysical. Could be Satellite Box or whitening electronics I suppose - I think we can exclude bad cabling, as that would just lead to the signals going to 0, whereas it would appear here that they did really change sign (confirmed by looking at the ULPDmon channel, which is digitized by Acromag, which reports -10 V at the time of glitch). But why should the BS care about the PRM electronics going wonky?
In addition to an exorcist, we need functioning electronics!
This optic has been hampering my locking attempts all evening. I switched the PRM and SRM satellite boxes, but then I remembered PRM has the Al foil "hats" to attenuate scattered light. of course the Al foil is conducting and can short the OSEM leads. I put some kapton pieces in between OSEM and foil to try and mitigate this issue but I suppose over time it could have slipped, and is making some intermittent contact, shorting PD anode and cathode (that would explain the PD reporting -10 V instead of some physical value).
If this is the problem we would need a vent to address it. In the daytime I'll measure L and R of the coils to see if the actuator imbalance I reported is also due to the same problem...
In preparation for later today evening. The TT alignment wasn't visibly disturbed.
Pity really, I was hoping to make it much further tonight. I think I'll have to go back to the high BW POX/POY lock, and also check out the conversion efficiency / noise of the daughter board on the REFL11 demod board. Compared to before my work on the RF source, the demod phase for the PRMI lock using REFL11 as an error signal has basically necessitated a change of the digital demod phase by 180 degrees - so I made the appropriate polarity changes in the CM_SLOW and AO paths (the assumption is that CARM in REFL11 would require the same change in digital demod phase, and I think this is a reasonable assumption - indeed, with the arm powers somewhat stable ~100, if I look at the PDH signal in REFL11 I and Q, it does seem to show up largely in the I quadrature (pre digital phase rotation). Anyway, with so many weird effects (wonky PRM suspension, strange PRMI sensing etc etc, who knows what's going on. This will take a systematic effort.
I defer the electronics characterization for the daytime (if I feel like I need it tomorrow I'll do it, else. Koji has said he can do it on Friday).
I was unable to fully hand off control from ALS-->RF, I suspect I may be using the wrong sign on the AO path (or some such other sub-optimal CM board settings). I'll hook up the SR785 and take some TFs tomorrow, that should give more insight into what's what.
The triggered code went on at 5:00 am today but a last minute change I made yesterday to increase number of repititions had an error and caused the script to exit putting everything back to normal. So as we came in the morning, we found the mode cleaner locked continuously after one free swing attempt at 5:00 am. I've fixed the script and ran it for 2 hours starting at 8;10 am. Our plan is to get some data atleast to play with when we are here. If the duration is not long enough, we'll try to run this again tomorrow morning. The new script is running on same tmux session 'MCFreeSwingTest' on Rossa
10:13 the script finished and IMC recovered lock.
Thu Mar 11 10:58:27 2021
The test ran succefully with the mode cleaner optics coming back to normal in the end of it. We wrote some scripts to read data and analyze it. More will come in future posts. No other changes were made today to the systems.
There is some evidence of weird saturation but the gain balancing (0.8dB) and orthogonality (~89 deg) for the daughter board on the REFL11 demod board that generates the AO path error signal seem reasonable. This board would probably benefit from the AD797-->Op27 and thick-film-->thin film swap but i don't think this is to blame for being unable to execute the RF transition.
I have recently been running into hitting the 4MB/s data rate limit on testpoints - basically, I can't run DTT TF and spectrum measurements that I was able to while locking the interferometer, which I certainly was able to this time last year. AFAIK, the major modification made was the addition of 4 DQ channels for the in-air BHD experiment - assuming the data is transmitted as double precision numbers, i estimate the additional load due to this change was ~500KB/s. Probably there is some compression so it is a bit more efficient (as this naive calc would suggest we can only record 32 channels and I counted 41 full rate channels in the model), but still, can't think of anything else that has changed. Anyway, I removed the unused parts and recompiled/re-installed the models (c1lsc and c1omc). Holding off on a restart until I decide I have the energy to recover the CDS system from the inevitable crash. For documentation I'm also attaching screenshot of the schematic of the changes made.
Anyway, the main point of this elog is that at the compilation stage, I got a warning I've never seen before:
Building front-end Linux kernel module c1lsc...
make: Warning: File 'GNUmakefile' has modification time 13 s in the future
make: warning: Clock skew detected. Your build may be incomplete.
This prompted me to check the system time on c1lsc and FB - you can see there is a 1 minute offset (it is not a delay in me issuing the command to the two machines)! I am suspecting this NTP action is the reason. So maybe a model reboot is in order. Sigh
Since Koji was in the lab I decided to bite the bullet and do the reboot. I've modified the reboot script - now, it prompts the user to confirm that the time recognized by the FEs are the same (use the IOP model's status screen, the GPSTIME is updated live on the upper right hand corner). So you would do sudo date --set="Thu 11 Mar 2021 06:48:30 PM UTC" for example, and then restart the IOP model. Why is this necessary? Who knows. It seems to be a deterministic way of getting things back up and running for now so we have to live with it. I will note that this was not a problem between 2017 and 2020 Oct, in which time I've run the reboot script >10 times without needing to take this step. But things change (for an as of yet unknown reason) and we must adapt. Once the IOPs all report a green "DC" status light on the CDS overview screen, you can let the script take you the rest of the way again.
The main point of this work was to relax the data rate on the c1lsc model, and this worked. It now registers ~3.2 MB/s, down from the ~3.8 MB/s earlier today. I can now measure 2 loop TFs simultaneously. This means that we should avoid adding any more DQ channels to the c1lsc model (without some adjustment/downsampling of others).
Holding off on a restart until I decide I have the energy to recover the CDS system from the inevitable crash.
I repeated the high bandwidth POY locking experiment.
One thing I am not sure is - when looking at the in-loop error point spectra, the Y-arm error point did not get suppressed to the CM board's sensing noise floor - I would have thought that with the huge amount of gain at ~16 Hz, the usual structure we see in the spectra between 10-30Hz would be completely squished. Need to think about if this is signalling something wrong, because the loop TF measurements seemed as expected to me.
1020pm: plots uploaded. As I made the plot of the spectrum, I realized that I don't have the calibration for the Y-arm error point into displacement noise units, so it's in unphysical units for now. But I think the comment about the hump around 16 Hz not being crushed to some sort of flat electronics noise floor. For the TF plots, when the loop gain is high, this IN1/IN2 technique isn't the best (due to saturation issues) but I don't think there's anything controversial about getting the UGF this way, and the fact that the phase evolves as expected when the various gains are cranked up / boosts enabled makes me think that the CM board is itself just fine.
10am 12 March: i realized that the "Y-arm error point" plotted below is not the true error point - that would be the input to the CM board (before boosts etc), which we don't monitor digitally. The spectra are plotted for the CM_SLOW input which already has some transfer function applied to it. In the past, I routed the LEMO "MON" connector on the demod board to the CM board input, and hence, had the usual SMA outputs from the demod board going to the digital system. I hypothesize that plotting the spectra for that signal would have showed this expected suppression to the electronics noise floor.
In summary, on the basis of this test, I don't see any red flags with the CM board.
For magnet strength measurement: There is a gaussmeter in the flukes' drawer (2nd from the top). It turns on and reacts to a whiteboard magnet.
I've brought 4 DO-32L-PE cards from WB for BHD upgrade for Jon.
I want to emphasize the followings:
I looked into this a bit today morning. I forgot exactly what time we restarted the machines, but looking at the timesyncd logs, it appears that the NTP synchronization is in fact working (log below is for c1sus, similar on other FEs):
-- Logs begin at Fri 2021-03-12 02:01:34 UTC, end at Fri 2021-03-12 19:01:55 UTC. --
Mar 12 02:01:36 c1sus systemd: Starting Network Time Synchronization...
Mar 12 02:01:37 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
Mar 12 02:01:37 c1sus systemd: Started Network Time Synchronization.
Mar 12 02:02:09 c1sus systemd-timesyncd: Using NTP server 192.168.113.201:123 (ntpserver).
So, the service is doing what it is supposed to (using the FB, 192.168.113.201, as the ntpserver). You can see that the timesync was done a couple of seconds after the machine booted up (validated against "uptime"). Then, the service is periodically correcting drifts. idk what it means that the time wasn't in sync when we check the time using timedatectl or similar. Anyway, like I said, I have successfully rebooted all the FEs without having to do this weird time adjustment >10 times.
I guess what I am saying is, I don't know what action is necessary for "implementing NTP synchronization properly", since the diagnostic logfile seems to indicate that it is doing what it is supposed to.
More worryingly, the time has already drifted in < 24 hours.
- Today we spent the morning shift debugging SUS input matrix diagonalization. MC stayed locked for most of the 4 hours we were here, and we didn't really touch any controls.
I may want to use the delay line phase shifter in 1Y2 to allow remote actuation of the REFL11 demod phase (for the AO path, not the low bandwidth one). I had this working last Feb, but today, I am unable to remotely change the delay. @Koji, it would be great if you could fix this the next time you are in the lab - I bet it's a busted latch IC or some such thing. I did the non-invasive tests - cable is connected, control bits are changing (at least according to the CDS BIO indicators) and the switch controlling remote/local switching is set correctly. The local switching works just fine.
In the meantime, I will keep trying - I am unconvinced we really need the delay line.
Attachment #1 - proof that the lock is RF only (A paths are ALS, B paths are RF).
Attachment #2 - CARM OLTF.
Some tuning can be done, the circulating power can be made ~twice as high with some ASC. The vertex is still on 3f control. I didn't get any major characterization done tonight but it's nice to be back here, a year on i guess.
On Friday, I felt that the ASC performance when the PRFPMI was locked was not as good as it used to be, so I looked into the situation a bit more. As part of my ASC model revamp in December, I made a bunch of changes to the signal routing, and my suspicion was that the control signals weren't even reaching the ETMs. My log says that I recompiled and reinstalled the c1rfm model (used to pipe the ASC control signals to the ETMs), and indeed, the file was modified on Dec 21. But for whatever reason, the C1RFM.ini (=Dolphin receiver since the ASC control signals are sent to this model over the Dolphin network from the c1ioo machine which hosts the C1:ASC- namespace, and RFM sender to the ETMs, but this path already existed) file never picked up the new channels. Today, I recompiled, re-installed, and restarted the models, and confirmed that the control signals actually make it to the ETMs. So now we can have the QPD-based ASC loops engaged once again for the PRFPMI lock. The CDS system did not crash 🎉 . See Attachments #1-3.
I checked the loop performance in the POX/POY locked config by first deliberately misaligning the ETMs, and then engagin the loops - seems to work (Attachment #4). The loop shapes have to be tweaked a bit and I didn't engage the integrators, hence the DC pointing wasn't recovered. Also, added a line to the script that turns the ASC loops on to set limits for all the loops - in the testing process, one of the loops ran away and I tripped the ETMY watchdog. It has since been recovered. I SDFed a limit of 100cts just to be on the conservative side for model reboot situations - the value in the script can be raised/lowered as deemed necessary (sorry, I don't know the cts-->urad number off the top of my head).
But the hope is this improves the power buildup, and provides stability so that I can begin to commission the AS WFS system a bit.
In the cleanroom, I opened the nickel-plated SmCo magnet box to take a closer look. I handled the magnets with tweezers. I wrapped the tips of the tweezers with some Kapton tape to prevent scratching and magnetization.
I put some magnets on a razor blade and took some close-up pictures of the face of the magnets on both sides. Most of them look like attachment 1.
Some have worn off plating on the edges. The most serious case is shown in attachment 2. Maybe it doesn't matter if we are going to sand them?
I measure the magnetic flux of the magnets by just attaching the gaussmeter flat head to the face of the magnet and move it around until the maximum value is reached.
For envelope #1 out of 3 the values are: (The magnet ordering is in attachment 3):
Going to continue tomorrow with the rest of the magnets. I left the magnet box and the gaussmeter under the flow bench in the cleanroom.
I'm going to remove REFL11 demod for the noise check/circuit improvement.
First I checked the noise levels and the transfer functions of the daughterboard preamp were checked. The CH-1 of the SR785 seemed funky (I can't comprehensively tell yet how it was), so the measurement maybe unreliable.
For the replacement of AD797, I tested OP27 and TLE2027. TLE2027 is similar to OP27, but slightly faster, less noisy, and better in various aspects.
The replacement of the AD797 and whatever-film resistors with LTE2027 and thin-film Rs were straightforward for the I phase channel, while the stabilization of the Q phase channel was a struggle (no matter I used OP27 or TLE2027). It seems that the 1st stage has some kind of instability and I suffered from 3Hz comb up to ~kHz. But the scope didn't show obvious 3Hz noise.
After a quite bit of struggle, I could tame this strange noise by adjusting the feedback capacitor of the 1st stage. The final transfer functions and the noise levels were measured. (To be analyzed later)
Now the REFL11 LO cable was replaced from the soft low noise audio coax (Belden 9239) to jacketed solder-soaked coax cable (Belden 1671J - RG405 compatible). The original cable indicated the delay of -34.3deg (@11MHz, 8.64ns) and the loss of 0.189dB.
I took 80-inch 1671J cable and measured the delay to be ~40deg. The length was adjusted using this number and the resulting cable indicated the delay of -34.0deg (@11MHz, 8.57ns) and the loss of 0.117dB.
The REFL11 demod module was restored and the cabling around REFL11 and AS110 were restored, tightened, and checked.
I've removed the PD mon cables from the NI RF switch. The open ports were plugged with 50Ohm temirnators.
Some progress today towards setting up an isolated subnet for testing the new front-ends. I was able to recover the fb1 backup disk using the Rescatux disk-rescue utility and successfully booted an fb1 clone on the subnet. This machine will function as the boot server and DAQ server for the front-ends under test. (None of these machines are connected to the Martian network or, currently, even the outside Internet.)
Despite the success with the framebuilder, front-ends cannot yet be booted locally because we are still missing the DHCP and FTP servers required for network booting. On the Martian net, these processes are running not on fb1 but on chiara. And to be able to compile and run models later in the testing, we will need the contents of the /opt/rtcds directory also hosted on chiara.
For these reasons, I think it will be easiest to create another clone of chiara to run on the subnet. There is a backup disk of chiara and I attempted to boot it on one of the LLO front-ends, but without success. The repair tool I used to recover the fb1 disk does not find a problem with the chiara disk. However, the chiara disk is an external USB drive, so I suspect there could be a compatibility problem with these old (~2010) machines. Some of them don't even recognize USB keyboards pre-boot-up. I may try booting the USB drive from a newer computer.
Edit: I removed one of the new, unused Supermicros from the 1Y2 rack and set it up in the test stand. This newer machine is able to boot the chiara USB disk without issue. Next time I'll continue with the networking setup.
Now that I think about it, I may only have backed up the root file system of chiara, and not/home/cds/ (symlinked to /opt/ over NFS). I think we never revived the rsync backup to LDAS after the FB fiasco of 2017, else that'd have been the most convenient way to get files. So you may have to resort to some other technique (e.g. configure the second network interface of the chiara clone to be on the martian network and copy over files to the local disk, and then disconnect the chiara clone from the martian network (if we really want to keep this test stand completely isolated from the existing CDS network) - the /home/cds/ directory is rather large IIRC, but with 2TB on the FB clone, you may be able to get everything needed to get the rtcds system working). It may then be necessary to hook up a separate disk to write frames to if you want to test that part of the system out.
Good to hear the backup disk was able to boot though!
And to be able to compile and run models later in the testing, we will need the contents of the /opt/rtcds directory also hosted on chiara.
For these reasons, I think it will be easiest to create another clone of chiara to run on the subnet. There is a backup disk of chiara and I attempted to boot it on one of the LLO front-ends, but without success.
After jumping through few hoops, we have one successful result in diagonalizing the input matrix for MC1, MC2 and MC3.
While Koji is working on the REFL 11 demod board, I took the opportunity to investigate the non-remote-controllability of the delay line in 1Y2, since the TTs have already been disturbed. Here is what I found today.
So it would seem something is not quite right with this BIO card. The c1lsc expansion chassis, in which this card sits, is notoriously finicky, and this delay line isn't very high priority, so I am deferring more invasive investigation to the next time the system crashes.
* I forgot we have the nice PCB Contec tester board with LEDs - the only downside is that this board has D37 connectors on both ends whereas the delay line wants a D25, necessitating some custom ribbon cable action. But maybe there is a way to use this.
As part of this work, I was in various sensitive areas (1Y3, chiara rack, FE test stand etc) but as far as I can tell, all systems are nominal.
I have added a .1", 45deg chamfer to the bottom of the adapter ring. This was added for a new placement of the eq stops, since the barrel screws are hard to access/adjust.
This also required a modification to the eq stop bracket, D960008-v2, with 1/4-20 screws angled at 45 deg to line up with the chamfer.
The issue I am running into is there needs to be a screw on the backside of the ring as well, otherwise the ring would fall backwards into the OSEMs in the event of an earthquake. The only two points of contact are these front two angled screws, a third is needed on the opposite side of the CoM for stability. This would require another bracket mounted at the back of the SOS tower, but there is very little open real estate because of the OSEMs.
Instead of this whole chamfer route, is it possible/easier to just swap the screws for the barrel eq stops? Instead of a socket head cap screw, a SS thumb screw such as this, will provide more torque when turning, and remove the need to use a hex wrench to turn.