I installed the Tripp Lite SMX1000RT2U and Tripp Lite Smart1000LCD at the bottom of the 1x8 electronics rack. These are plugged in to power, and are ready for testing. All other cables (serial, usb, etc.) have been left on the table next to the 1x8 rack.
This year we've struggled with vacuum controls unreliability (e.g., spurious interlock triggers) caused by decaying hardware. Here are details of the vacuum refurbishment plan I described on the 40m call this week.
☑ Refurbish TP2 and TP3 dry pumps. Completed [ELOG 15417].
☑ Automated notifications of interlock-trigger events. Email to 40m list and a new interlock flag channel. Completed [ELOG 15424].
☐ Replace failing UPS.
☐ Remove interlock dependencies on TP2/TP3 serial readbacks. Due to persistent glitching [ELOG 15140, ELOG 15392].
Unlike TP2 and TP3, the TP1 readbacks are real analog signals routed to Acromags. As these have caused us no issues at all, the plan is to eliminate dependence on the TP2/3 digital readbacks in favor of the analog controller outputs. All the digital readback channels will continue to exist, but the interlock system will no longer depend on them. This will require adding 2 new sinking BI channels each for TP2 and TP3 (for a total of 4 new channels). We have 8 open Acromag XT1111 channels in the c1vac system [ELOG 14493], so the new channels can be accommodated. The below table summarizes the proposed changes.
I decided to use the old EY auxiliary optics table, which is now stored along the east arm about 10 m from the end, as a workspace for assembling the little PMCs. I wiped everything down with isopropanol for general cleanliness, removed the metal plate on the south edge of the table enclosure to allow access, covered the table with some clean Aluminium foil, and then moved the plastic box with PMC parts to the table - see Attachment #1. I haven't actually done any assembly just yet, waiting for more info (if available) on the procedure and implements available...
Attachment #1 shows the RIN of the local oscillator beam delivered to the AP table via fiber. I used a PDA520 to make this measurement, while the electronics for the DCPDs are pending. I don't really have an explanation for the difference between the locked IFO trace vs the not locked trace - we don't have an ISS running (but this first test suggests we should) and the beam is picked off before any cavities etc, so this is a reflection of the state of the FSS servo at the times of measurement?
Tried locking CARM using the hybrid REFL (for AO path) and POX 11 (for MCL path) scheme a bunch of times today, but I had no luck. When the CARM offset is zeroed, the PRMI lock is lost almost immediately. Maybe this is indicative of some excess noise in the POX data stream relative to the REFL signal? The one thing I haven't tried is to take the IFO all the way to the locked state, and then transition the MCL actuation from CM_SLOW to POX11_I.
An SR785 is sitting on the North side of the AP table in the walkway - I will clear it tomorrow.
I've added 4 proposed schemes for implementing ALS in voyager. Major thing to figure out is what AUX laser would be and how we would compare the different PSL and AUX lasers to create an error signal for ALS. Everywhere below, 2um would mean wavelengths near 2 um including the proposed 2128nm. Since it is not fixed, I'm using a categorical name. Same is the case for 1um which actually would mean half of whatever 2 um carries.
The LO pickoff has been coupled into a fiber with ~90% MM (8 mW / 9 mW input). While I wait for the DCPD electronics to be found in the Cryo lab, I want to monitor the stability of the pointing, polarization etc, so I'd like to clear some space on the AP table that was occupied for the mode spectroscopy project. If there are no objections before 2pm tomorrow July 21 2020, I will commence this work.
Looking at the signals to the test mass coils, it seems borderline to me that we will be able to acquire lock and run in a low noise configuration with the same series resistor in the coil driver circuit. The way I see it, options are:
I only looked at the ETMs for this study. The assumption is that we will have no length actuation on the ITMs, only local damping and Oplev loops (and maybe some ASC actuation?), which can be sufficiently low-pass filtered such that even with coil de-whitening, we won't have any range issues.
Attachment #1 shows the time-domain traces of the coil driver signals as we transition from POX/POY lock to the ALS lock. There are some transients, but I think we will be able to hold the lock even with a 5 kohm resistor (~twice what is on ETMX right now). From just these numbers, it would seem we can even go up to 10 kohms right away and still be able to acquire lock, especially if we re-design the digital feedback loop to have better low-pass filtering of the high-frequency ALS noise, see the next attachment.
Attachment #2 shows the f-domain picture, once the arm lengths are fully under ALS control (~25 seconds onwards in Attachment #1). The RMS is dominated by high frequency ALS length loop noise, which we can possibly improve with better design of the digital control loop.
Finally, Attachment #3 shows the situation once DARM control has been transitioned over to AS55_Q. Note that the vertex DoFs are still under 3f control, so there is the possibility that we can make this even lower noise. However, one thing that is not factored in here is that we will have to de-whiten these signals to low-pass filter the DAC noise (unless there is some demonstrated clever technique with noise-mons or something to subtract the DAC noise digitally). Nevertheless, it seems like we can run safely with 5 kohms on each ETM coil and still only use ~2000 cts RMS, which is ~1/10th the DAC range (to allow for dealing with spurious transients etc).
Looking at signals to the ETMs from the current lock acquisition sequence, the RMS current to a single coil is approximately _____ (to be filled in later).
Attachment #1 shows the proposed wiring and CDS topology for the in air BHD setup. The PDF document has hyperlinks you can follow to the DCC entries. Main points:
Please comment if I've overlooked something.
I will be in the Clean and Bake lab today from 9am to 4pm.
The DQ channels of the ETM coils were active tonight, so I'll make the coil driver actuation budget over the next couple of days.
The (masked) tech accessed all areas in the lab (office area, control room, VEA) between ~230pm-3pm. The laser safety goggles he used have been kept aside for appropriate sanitaiton.
Attachment #1 - The 80mW pickoff was getting clipped on a BNC cable, and not making it to the doubling oven. 😢 .
Attachment #2 - PSL green shutter removed. Alignment into the doubling oven is extremely tedious, and so I opted to preserve the capability of recovering the green beam by simply removing a single mirror.
Attachment #3 - The beam path for coupling the LO beam into a fiber.
Attachment #4 shows the BHD photodiodes taken from QIL.
To facilitate this investigation, I've DQed the 4 face coil outputs for the two ETMs. EX is currently running with 5 times the series resistance of EY, so it'll be a nice consistency check. Compilation, installation etc went smooth. But when restarting the c1scx model, there was a weird issue - the foton file, C1SCX.txt, got completely wiped (all filter coefficients were empty, even though the filter module names themselves existed). I just copied the chiara backup version, restarted the model, and all was well again.
This corresponds to 8 additional channels, recorded at 16k as float 32 numbers, so in the worst case (neglecting any clever compression algorithms), we are using disk space at a rate of ~4 MB/s more. Seems okay, but anyway, I will remove these DQ channels in a few days, once we're happy we have enough info to inform the coil driver design.
spoke too soon - there was an RFM error for the TRX channel, and restarting that model on c1sus took down all the vertex FEs. Anyways, now, things are back to normal I think. The remaining red light in c1lsc is from the DNN model not running - I forgot to remove those channels, this would've been a good chance! Anyways, given that there is an MLTI in construction, I'm removing these channels from the c1lsc model, so the next time we restart, the changes will be propagated.
For whatever reason, my usual locking scripts aren't able to get me to the PRFPMI locked state - some EPICS channel value must not have been set correctly after the model reboot 😞. I'll debug in the coming days.
Fun times lie ahead for getting the new BHD FEs installed I guess 🤡 ....
So we may need a version of the fast coil driver that supports a low noise mode (with large series resistance) and a high-range mode (with lower series resistance for lock acquisition).
True - it is now not on anymore.
It happened before too. Doesn't it say it has occasional self-testing or something?
The emergency lamps above the exit sign on the NW entrance to the control room are on. I tried opening and closing the door, but it remains on. Probably nothing to worry about, but noting here anyway.
You can activate all 3axis
I forgot about the pointing - probably we will need another actuator to control the pointing of the AS beam onto the DCPDs. I found a few old PI PZTs (model number is S-320, which is a retired part), one is labelled broken but the others don't indicate a-priori that they are broken. I'll post a more detailed hardware survey later.
I started my attempt on noise budgeting of ALS by going back to how Kiwamu did it and adding as many sources as I could find up till now. This calculation is present in ALS_Noise_Budget notebook. I intend to collect data for noise sources and all future work on ALS in the ALS repo.
The noise budget runs simulink through matlab.engine inside python and remaining calculations including the pygwinc ones are done in python. Please point out any errors that I might have done here. I still need to add noise due to DFD and the ADC after it. For the residual frequency noise of AUX laser, I have currently used an upper limit of 1kHz/rt Hz at 10 Hz free-running frequency noise of an NPRO laser.
From Attachment #1, looks like the phasing and gain for CARM on POX11 is nearly the same as CARM of REFL11, which is probably why I was able to execute a partial transition last night. The response in POY11 is ~10 times greater than POX11, as expected - though the two photodiodes have similar RF transimpedance, there is a ZFL-500-HLN at the POY11 output. The actual numerical values are 2.5e10 cts/m for CARM-->REFL11_I, 2.6e10 cts/m for CARM-->POX11_I, and 3.2e11 cts/m for CARM-->POY11_I.
So I think I'll just have to fiddle around with the transition settings a little more tonight.
One possible concern is that the POX and POY signals are digitized without preamplificatio, maybe this explains the larger uncertainty ellipse for the POX and POY photodiodes relative to the REFL11 photodiode? Maybe the high frequency noise is worse and is injecting junk in the AO path? I think it's valid to directly compare the POX and REFL spectra in Attachment #2, without correcting for any loops, because this signal is digitized from the LSC demodulator board output (not the preamplified one, which is what goes to the CM board, and hence, is suppressed by the CARM loop). Hard to be sure though, because while the heads are supposed to have similar transimpedance, and the POX photodiode has +12dB more whitening gain than REFL11, and I don't know what the relative light levels on these photodiodes are in lock.
I have some data from a couple of days ago when the PRFPMI was locked as usual (CARM_B on REFL for both DC and AO paths), and the sensing lines were on, so I can measure the relative strength of the sensing lines in POX/REFL and get an estimate of what the correct digital gain should be
Koji and I had a discussion last Friday about the suspension electronics. I think there are still a few open questions - see Attachment #1. We should probably make a decision on these soon.
Other useful links:
For the first pass, it's probably easiest to use the existing DCPD amplifier. Looking at the gain and noise performance in Attachment #1, seems totally fine, the electronics noise will not be limiting if we have ~10mW of LO power. I assumed a transimpedance resistor of 1 kohm, and all other numbers as on the schematic (though who knows if the schematic is accurate). The noise should be measured to confirm that the box is performing as expected...
I will be in the Clean and Bake Lab today from 9am to 4pm.
The usual technique is that keeping the IFO locked with the old set of the signals and the relative gain/TF between the conventional and new signals are measured in-lock so that you can calibrate the new gain/demod-phase setting.
I tried using the POX_I error signal for the DC CARM_B path today a couple of times. Got to a point where the AO path could be engaged and the arm powers stabilized somewhat, but I couldn't turn the CARM_A path off without blowing the lock. Now the IMC has entered a temperemental state, so I'm abandoning efforts for tonight, but things to try tomorrow are:
I have some data from a couple of days ago when the PRFPMI was locked as usual (CARM_B on REFL for both DC and AO paths), and the sensing lines were on, so I can measure the relative strength of the sensing lines in POX/REFL and get an estimate of what the correct digital gain should be.
The motivation here is to see if the sensing matrix looks any different with a modified locking scheme.
In fact, all these utilities are now available in python3. There may be some bugs (e.g. this), but I've checked basic functionality and things look usable enough for development to proceed. While we can have a python2 env on rossa, I think it's unnecessary.
I too, would prefer py3 for everything, but aren't all the cdsutils / gaurdian things still python2?
Is it possible to just make a python2 conda environment on rossa? I would guess that its simple and won't interfere with the regular operation of that machine.
After some hunting, I found this old SURF report with the WFS head measurements. The y-axes don't make much sense to me, and I can't find the actual data anywhere (her wiki page doesn't actually exist). So I think it's still unknown if these heads ever had the advertised transimpedance gain, or if the measured transimpedance of ~1kohm was what it always was.
Main goals tonight were:
> Can't we offload this DC signal to the laser crystal temperature servo?
No. PSL already follows the MC length. So this offset is coming from the difference between the MC length and the CARM length.
What you can do is to offload the MC length to the CARM DC if this helps.
After some consultation with Erik von Reis at LHO, this workstation is progressing towards being usable for most commissioning tasks. DTT, awggui, foton, and MEDM are all now working well. The main limitation now comes from the fact that many of our python scripts are written for python2, and rossa doesn't have many dependencies installed for python2. I see no reason to build these dependencies on rossa for python2, we should not have to work with an unsupported language. But at the same time, I don't want to completely wipe all our python2 scripts, and make them python3, because this would involve a lot of tedious testing that I'm not prepared to undertake at the moment (the problem is compounded by the fact that pianosa does not have many dependencies installed for python3).
So what I have done in the interim is make python3 versions of the most important scripts I need to get the PRFPMI locking working - they are in the scripts directory and have the same names as their python2 counterparts, but have a 3 appended to their names. So when working on rossa, these are the scripts that are called. Eventually, after a lot more testing, we can depracate the old scripts. Currently, where applicable, the MEDM screens allow for either the python2 or python3 version of the script to be called.
Please, for the time being, do not try and install any new packages on rossa unless you are prepared to debug any problems caused and return the machine to a workable state. If you find some issue with a missing package on rossa, (i) make a note of it on the elog, and (ii) if possible, set up your own conda environment for testing and install dependencies to that environment only.
I was looking at some signals from last night, see Attachment #1.
Attachment #2 shows some ASC metrics. My conclusion here is that running the PRCL and MICH dither alignment servos (former demodulating REFLDC and latter demodulating ASDC to get an error signal) that running the dither alignment servo and hand tuning the arm ASC loop offsets improves the mode matching to the IFO, because:
The REFLDC behavior needs a bit more interpretation I think, because if the IFO is overcoupled (as I claim it is), then better alignment would at some point actually result in REFLDC increasing.
All the DC signals recorded by the fast system come from the backplane P2 connector of the PD interface boards. According to the schematic, these signals have a voltage gain of 2. The LSC photodiodes themselves have a nominal DC gain of 50 ohms. So, the conversion from power to digital counts is: 0.8 A/W * 50 V/A * 2 * 3276.8 cts/V * whtGain. Inverting, I get 3.8 uW/ct for a whitening gain of 1. This is power measured at the photodiode - optical losses upstream of the photodiode will have to be accounted for separately.
Assuming a modulation depth of 0.2, the 55 MHz sideband power should be ~20 mW. The Schnupp asymmetry is supposed to give us O(1) transmission of this field to the AS port. Then, the SRM will attenuate the field by a factor of 10, so we expect ~2 mW at the AS port. Let's assume 80 % throughput of this field to the AP table, and then there is a 50/50 beamsplitter dividing the light between the AS55 and AS110 photodiodes. So, we expect there to be ~700 uW of power in the TEM00 mode 55 MHz sideband field. This corresponds to 1600 cts according to the above calibration (the ASDC whitening gain is set to 18 dB). The fact that much smaller numbers were seen for ASDC indicates that (i) the schnupp asymmetry is not so perfectly tuned and the actual transmission of the sideband field to the dark port is smaller, or (ii) one or more optical splitting fractions assumed above is wrong. If the former is true, we can still probably infer the contrast defect if we can somehow get an accurate measurement of the sideband transmission to the dark port.
I will be in the Clean and Bake lab today from 9am to 4pm
More tomorrow, but I tried the following tonight:
Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).
They will arrive within the next two weeks.
I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:
I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.
We can probably learn something about the interferometer / top level BHD plan with an in-air BHD setup, even if the noise is bad. Here are some thoughts about how we would do it.
For this first attempt, we don't really care about the PRC filtering. So possible places to pick off an LO beam are:
In all cases, I think the easiest option to actually route whatever beam we choose into a fiber, and then bring it over to whatever cavity we choose to use for an OMC. I'm assuming whatever phase control technique we end up using can cancel the fiber phase noise at relevant frequencies.
LO phase control
There is a question about the range, but I think these are the only two realistic options we can implement on a reasonable time scale.
Again, there are a few options. Here are some pros and cons that come to my mind.
If we can do a vent (we'd just need a single chamber open), I'd go for the option of getting the copper OMC out and using that. Attachment #1 shows the approximate sizes of the various components (OMMT, OMC cavity, DCPDs), while Attachment #2 shows a rough sketch of where things would go on the AP table, with the rectangles approximately to scale.
I'd made a c1omc model sometime ago. Basically, I think we have sufficient ADC/DAC channels in the c1ioo machine for any of the options listed above - but using the copper OMC and associated peripherals would allow the easiest interfacing.
I noticed these streaky lines again today (but they were not a problem last night). It is annoying if we have to reboot this machine all the time. I wonder if this has something to do with missing drivers. When I ran sudo apt update && sudo apt upgrade, I got several lines like (this isn't the whole stack trace)
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/ucode_unload.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/ucode_load.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/unload_bl.bin for module nouveau
W: Possible missing firmware /lib/firmware/nvidia/gp108/acr/bl.bin for module nouveau
Is this indicative of the graphics drivers being installed incorrectly? I am hesitant to mess with this because I think in the past, it was always trying to update some graphics driver that crashed the whole machine into some weird state where we have to wipe the drive and do a fresh re-install of the OS.
Should we just follow these instructions? The graphics card is apparently Quadro P400, which is one of the supported ones according to the list of supported devices.
Or just swap donatella and rossa monitors and defer the problem for later?
yes, I rebooted yesterday to fix the 'steaking white lines' problem in the video/display
Here is the procedure for setting up the three new BHD front-ends (c1bhd, c1sus2, c1ioo - replacement). This plan is based on technical advice from Rolf Bork and Keith Thorne.
The overall topology for each machine is shown here. As all our existing front-ends use (obsolete) Dolphin PCIe Gen1 cards for IPC, we have elected to re-use Dolphin Gen1 cards removed from the sites. Different PCIe generations of Dolphin cards cannot be mixed, so the only alternative would be to upgrade every 40m machine. However the drivers for these Gen1 Dolphin cards were last updated in 2016. Consequently, they do not support the latest Linux kernel (4.x) which forces us to install a near-obsolete OS for compatibility (Debian 8).
I will be in the Clean and Bake lab today from 9am to 3pm
I wanted to try using rossa as my locking workstation today. However, a few problems became quickly evident. Basically, any of our scripts that rely on the cdsutils package (there are MANY) will not work on rossa, because of some library error. This machine is running Debian 10, while the cdsutils package is being loaded from a pre-compiled install on the shared drive, so perhaps this isn't surprising?
Digging a little more, I found that actually, a version of cdsutils that actually works with python3 is actually shipped with the standard cds-workstation meta-package. This is great news, and we should try and use this where possible I guess. Deferring further debugging for daytime work.
Anyway, I added a symlink: sudo ln -s /usr/lib/x86_64-linux-gnu/libncurses.so.6 /usr/lib/x86_64-linux-gnu/libncurses.so.5, and installed wmctrl using sudo apt install wmctrl.
I will be in the clean and bake lab today from 9am to 3pm.
I want to be able to run the dither alignment servo with the PRFPMI locked - I've been thinking about what the scheme should be, and I list here some questions I had while thinking about this.
Last Tuesday evening, while attempting the PRFPMI locking, I noticed a strange feature in the LSC signals, which is shown in Attachment #1 (the PDF exported by dataviewer is 14MB so I upload the jpeg instead). As best as I can tell, the REFL33 and POP22 channels show an abrupt jump in the signal levels, while the other channels do not. POP110 shows a slight jump at around the same time, and the large excursion in AS110_Q actually occurs a few seconds later, and is probably some angular excursion of the PRC/BS. I'm struggling to interpret how this can be explained by some interferometric mechanism, but haven't come up with anything yet. The LO for the 3f error signals is the 2f field, but then why doesn't the POP110 channel show a similar jump if there is some abrupt change in the resonant condition? Is such a change even feasible from a cavity length change point of view? Or did the sideband frequency somehow abruptly jump? But if so, why is the jump much more clearly visible in one sideband than the other?
Does anyone have any ideas as to what could be going on here? This may give some clue as to what's up with the weird sensing matrices, but may also be something boring like broken electronics...
As suggested last week, Hang and I have reviewed the A+ BHD status (DRD, CDD, and reviewers' comments) and compiled a list of key unanswered questions which could be addressed through Finesse analysis.
In anticipation of others helping with this modeling effort, we've tried to break questions into self-contained projects and estimated their level of difficulty. As you'll see, they range from beginner to Finesse guru.
Indeed, this is now fixed by following instructions from here. I rebooted rossa at ~1250 PDT and confirmed that resolv.conf didn't get overwritten. The resolv.conf file also now has the following useful lines at the head:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
maybe we're supposed to edit something besides resolv.conf since that gets over-written on boot for some linux OS
I will be in the Clean and Bake lab today from 8:30am to 4pm
This is strange - I was definitely able to launch medm when I was working on this machine remotely on Friday. But now, there does seem to be a problem with this shared library being missing.
First of all, I installed mlocate to find where the shared library files are installed. Then I made the symlink, and now sitemap seems to work again.
Weirdly, my changes to /etc/resolv.conf got overwritten somehow. Was this machine rebooted? Uptime suggests it's only been running for ~6 hours at the time of writing of this elog.
sudo apt install mlocate
sudo ln -s /usr/lib/x86_64-linux-gnu/libreadline.so.7 /usr/lib/x86_64-linux-gnu/libreadline.so.6
when I try 'sitemap' on rossa I get:
medm: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory
sudo usermod -a -G lpadmin controls
and then was able to add Grazia to the list of printers for Rossa by following the instructions on the 40m Wiki.
I installed color syntax highlighting on Rossa using the internet (https://superuser.com/questions/71588/how-to-syntax-highlight-via-less). Now if you do 'less genius_code.py', it will be highlighting the python syntax.
medm: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory