Bulb replaced. Projector is back on.
The control room UPS started making a beeping noise saying batteries need replacement. I hit the "Test" button and the beeping went away. According to the label on it, the batteries were last repalced in March 2016, so maybe it is time for a replacement, @Chub, please look into this.
For some reason, rossa's Xdisplay won't start up anymore. This happened right after the UPS reset. Koji and I tried ~1.5 hours of debugging, got nowhere.
The database files for C1ISCAUX seem to work file - the exception being the mbbo channels for the CM board.
This was just a software test - the actual functionality of the channels will have to be tested once the Acromag crate has been installed in the rack. One change I had to make on the MEDM screen for the LSC PD whitening gains was to get rid of the "NMS" suffix on the EPICS channel names for whitening gain sliders/drop-down-menus. I suspect this has to do with the EPICS version we are using, 7.0.1. Furthermore, AS165 and POP55 no longer exist - I hold off removing them from the MEDM screen for the moment.
From the software point of view, the major steps are:
I am stopping the EPICS server on the new machine and restarting the old VME crate over the weekend.
What channels are you trying to read?
I'm not able to get trends of the TM adjustment test that Rana had asked us to perform, from the dataviewer. It's throwing the following error:
Connecting to NDS Server fb (TCP port 8088)
Server error 7: connect() failed
datasrv: DataWrite failed: daq_send: Resource temporarily unavailable
T0=19-07-20-01-27-39; Length=600 (s)
No data output.
SnapPy scripts made to work on Pianosa.
Of course rossa was the only machine in the lab that could run the python scripts to interface with the GigE camera. And it is totally bricked now. Lame.
So I installed several packages. The key was to install pypylon - if you go to the basler webpage, pypylon1.4.0 does not offer python2.7 support for x86_64 architecture, so I installed pypylon1.3.0. Here are the relevant lines from the changelog:
gstreamer-plugins-bad-0.10.23-5.el7.x86_64 Sat 20 Jul 2019 11:22:21 AM PDT
gstreamer-plugins-good-0.10.31-13.el7.x86_64 Sat 20 Jul 2019 11:22:11 AM PDT
gstreamer-plugins-ugly-0.10.19-31.el7.x86_64 Sat 20 Jul 2019 11:20:08 AM PDT
gstreamer-python-devel-0.10.22-6.el7.x86_64 Sat 20 Jul 2019 10:34:35 AM PDT
pygtk2-devel-2.24.0-9.el7.x86_64 Sat 20 Jul 2019 10:34:34 AM PDT
pygobject2-devel-2.28.6-11.el7.x86_64 Sat 20 Jul 2019 10:34:33 AM PDT
pygobject2-codegen-2.28.6-11.el7.x86_64 Sat 20 Jul 2019 10:34:33 AM PDT
gstreamer-devel-0.10.36-7.el7.x86_64 Sat 20 Jul 2019 10:34:32 AM PDT
gstreamer-python-0.10.22-6.el7.x86_64 Sat 20 Jul 2019 10:34:31 AM PDT
gtk2-devel-2.24.31-1.el7.x86_64 Sat 20 Jul 2019 10:34:30 AM PDT
libXrandr-devel-1.5.1-2.el7.x86_64 Sat 20 Jul 2019 10:34:28 AM PDT
pango-devel-1.42.4-1.el7.x86_64 Sat 20 Jul 2019 10:34:27 AM PDT
harfbuzz-devel-1.7.5-2.el7.x86_64 Sat 20 Jul 2019 10:34:26 AM PDT
graphite2-devel-1.3.10-1.el7_3.x86_64 Sat 20 Jul 2019 10:34:26 AM PDT
pycairo-devel-1.8.10-8.el7.x86_64 Sat 20 Jul 2019 10:34:25 AM PDT
cairo-devel-1.15.12-3.el7.x86_64 Sat 20 Jul 2019 10:34:25 AM PDT
mesa-libEGL-devel-18.0.5-3.el7.x86_64 Sat 20 Jul 2019 10:34:24 AM PDT
libXi-devel-1.7.9-1.el7.x86_64 Sat 20 Jul 2019 10:34:24 AM PDT
pygtk2-doc-2.24.0-9.el7.noarch Sat 20 Jul 2019 10:34:23 AM PDT
atk-devel-2.28.1-1.el7.x86_64 Sat 20 Jul 2019 10:34:21 AM PDT
libXcursor-devel-1.1.15-1.el7.x86_64 Sat 20 Jul 2019 10:34:20 AM PDT
fribidi-devel-1.0.2-1.el7.x86_64 Sat 20 Jul 2019 10:34:20 AM PDT
pixman-devel-0.34.0-1.el7.x86_64 Sat 20 Jul 2019 10:34:19 AM PDT
libXinerama-devel-1.1.3-2.1.el7.x86_64 Sat 20 Jul 2019 10:34:19 AM PDT
libXcomposite-devel-0.4.4-4.1.el7.x86_64 Sat 20 Jul 2019 10:34:19 AM PDT
libicu-devel-50.1.2-15.el7.x86_64 Sat 20 Jul 2019 10:34:18 AM PDT
gdk-pixbuf2-devel-2.36.12-3.el7.x86_64 Sat 20 Jul 2019 10:34:17 AM PDT
pygobject2-doc-2.28.6-11.el7.x86_64 Sat 20 Jul 2019 10:34:16 AM PDT
pygtk2-codegen-2.24.0-9.el7.x86_64 Sat 20 Jul 2019 10:34:15 AM PDT
Camera server is running on a tmux session on pianosa. But it keeps throwing up some gstreamer warnings/errors, and periodically (~every 20 mins) crashes. Kruthi tells me that this behavior was seen on Rossa as well, so whatever the problem is, doesn't seem to be because I missed out on installing some packages on pianosa. Moreover, if the server is in fact running, I am able to take a snapshot - but the camera client does not run.
The boards arrived. I soldered on a DIN96 connector, and tested that the goemetry will work. It does . The only constraint is that the P2 interface board has to be installed before the P1 interface is installed. Next step is to confirm that the pin-mapping is correct. The pin mapping from the DIN96 connector to the DB15 was also verified.
*Maybe it isn't obvious from the picture, but there shouldn't be any space constraint even with the DB37/DB15 cables connected to the respective adapter boards.
Can you please be more specific about what the error is? Is this the usual instability with the camera server code? Or was it something new?
The camera server is throwing an error and is not grabbing snapshots :(
DATED, SEE ELOG14941 for the most up-to-date info on latch.py.
I wrote (/cvs/cds/caltech/target/c1iscaux3/latch.py) and tested the logic illustrated in Attachment #1. Results of a test are shown in Attachment #2, the various channels change as expected. Note that for negative values of the gain channel, the corresponding "BITS" channel will take on values like 65536 - this is because the mbboDirect data type is a 16 bit data type, and presumably the MSB is the sign bit. A bit mask is applied to this channel before the actual BIO unit bits are set - we should verify that the correct behavior happens, but I don't immediately see any problems.
To me, this is a robust logic, but it will benefit from more sets of eyes giving it a look over. The idea is to run this continuously on the Supermicro machine.
Apart from this, I also fixed some errors in the mbboDirect record syntax - so now I am able to start up the EPICS server without it throwing any error messages. It remains to verify that changing an EPICS gain slider results in the appropriate gain bits being flipped in the correct way (on the hardware side, I think the correct behavior is happening on the software end). For this testing, I turned off the old c1iscaux crate at ~10am, and started up the server on c1iscaux3. I am reverting to the nominal config now (~1pm).
Further testing will require the wiring inside the Acromag chassis to be completed. This should be the priority task for next week.
*Update 1130 22 July 2019: I've now installed the required dependencies on c1iscaux3 and setup the latch.py script to run as a systemctl process dependent on modbusIOC.service.
Somebody changed the settings on painosa without elogging anything about it. Why does this keep happening? I thought the point of the elog was to communicate. I think there are sufficient number of problems in the lab without me having to manually reset the control room workstation settings every week. Please make an elog if you change something.
My goal tonight was to lock the PSL frequency to be resonant in the XARM cavity, using the PSL+EX beat as the error signal. I was not successful - mainly, I was plagued by huge BR mode coupling in the error signal, and I could not enable the BR notch filter in the control loop without breaking the lock. Need to think about next steps.
Anyway, now that I have a workable set of settings that gets me close to the ALS lock of the XARM, I expect debugging to proceed faster.
Update 2019 July 23: I looked at the control loop shape today, see Attachment #3. I'm not sure I understand why the "BounceRoll" filter in this filter bank looks like a resonant gain rather than a notch, as it does for the Oplev or SUSPOS loops for example - don't we want to not actuate at these frequencies because the reason the signal exists is because of the imperfect OSEM/magnet positioning? This does not explain the spectrum shown in Attachment #2 however, as that filter was disabled.
I succeeded in locking the PSL frequency to the XARM cavity length, with 9 pm RMS (Attachment #1) motion below 1 kHz, by actuating on MC2 to change the IMC length. The locks were pretty stable (~20 minutes) - the dominant cause of lockloss was the infamous ETMX drifting problem.
My main motivation here is to make some measurements and investigate the SoCal idea using a toy system, i.e. a single arm cavity controlled using ALS, so that's what Craig and I will attempt next.
Since there are multiple SURF projects that rely on the cameras:
My changes were necessary because the grabHDR.py script was throwing python exceptions, whereas it was running just fine before Jon's changes. We can move the "new_*" dirs to the default once the SURFs are gone.
Let's freeze the camera software config in this state until next week.
These spectra were taken with the arm cavity length locked to the PSL frequency using POX as an error signal, and the EX laser frequency locked to the XARM cavity length by the analog PDH servo at EX, so there is no feedback control with the ALS beat signal as an error signal.
I think rana did some more changes to this workstation to make it useful for commissioning activities - but the MEDM screens were still white blanks. The problem was that the firewalld wasn't disabled (last two steps of the KThorne setup wiki). I disabled it. Now donatella can run MEDM, ndscope and StripTool. DTT doesn't work to get online data because of a "Synchronization Error", I'm not bothering with this for now. I think Kruthi successfully demonstrated the fetching of offline data with DTT.
I brought one CPU (Dell T3500) and one 28" monitor from Mike Pedraza's office in Downs to the 40m. It is on Steve's desk right now, pending setup. The machine already has Solidworks and Altium installed on it, so we can set it up at our leisure. The login credentials are pasted on the CPU with a post-it should anyone wish to set it up.
We need to determine the geometry (= round-trip length and RoC of curved mirrors) of the OMC cavities for the 40m BHD experiment. Sticking to the aLIGO design of a 4 mirror bowite cavity with 2 flat mirrors and 2 curved mirrors, with a ~4deg angle of incidence, we need to modify the parameters for the 40m slightly on account of our different modulation frequencies. I've setup some infrastructure to do this analytically - even if we end up doing this with Finesse, it is useful to have an analytic calculation to validate against (also not sure if Finesse can calculate HOMs up to order 20 in a reasonable time, I've only seen maxtem 8).
Attachment #1: Heatmap of the OMC transmission for the following fields:
The code used for the ABCD matrix calcs have been uploaded to the BHD modeling GIT (but not the one for making this plot, yet, I need to clean it up a bit). Some design considerations have also been added to our laundry list on the 40m wiki.
Chub brought the replacement Supermicro we ordered to the 40m today. I stored it at the SW entrance to the VEA, along with the other Supermicro. At the time of writing, we have, in hand, two (unused) Supermicro machines. One is meant for EY and the other is meant for c1psl/c1iool0. DDR3 RAM and 120 GB SSD drives have also been ordered, but have not yet arrived (I think, Chub, please correct me if I'm wrong).
Update 20190802: The DDR3 RAM and 120 GB SSD drives arrived, and are stored in the FE hardware cabinet along the east arm. So at the time of writing, we have 2 sets of (Supermicro + 120GB HD + 4GB RAM).
We should ask Chub to reorder several more SuperMicro rackmount machines, SSD drives, and DRAM cards. Gautam has the list of parts from Johannes' last order.
I'd like to confirm that the IR ALS scheme will work for locking. The X-arm performance so far has been encouraging. I want to repeat the characterization for the Y arm. So I inspected the layout on the EY table, and made a list of characterization tasks. The current EY beam routing is difficult to work with, and it will definitely benefit from a re-do. However, I don't know how much time I want to spend re-doing it, so for a start, I will just try and couple some amount of light into a fiber and bring it to the PSL table, and see what noise performance I get.
Attachment #1: Photo of the current beam layout. The powers indicated were measured with the Ophir power meter.
Attachment #2: A candidate mode-matching solution, given the constraints outlined above. It isn't great, with only 85% modematching even theoretically possible. The lenses required are also pretty fast lenses. But I think it's the best possible without a complete overhaul of the EY layout. I'm still waiting for the lens kit to arrive, but as soon as they get here, I will start this work.
I want to collect some data with the arms locked to investigate the possibility/usefullness of having seismic feedforward implemented for the arms (it is already known to help the IMC length and PRC angular stability at low frequencies). To facilitate diagnostics I modified the file /users/Templates/Seismic/Seismic_vs_TRXTRYandMC.xml to have the correct channel names in light of Lydia's channel name changes in 2016. Looking at the coherence data, the alignment of the cartesian coordinate system of the Seismometers at the ends and the global interferometer coordinate system can be improved.
I don't know if for the MISO filter design if there is any difference in using TRX/TRY as the target, or the arm length control signal.
Data collection started at 1249018179. I've setup a script running in a tmux shell to turn off the LSC enable in 2 hours.
The VEA laptop asia was configured to be able to connect to too many WiFi networks - it was getting conflicted in its default position at the vertex and trying to hop between networks, for some reason trying to connect to networks that had poor signal strength. I deleted all options from the known networks except 40MARS. Now the network connection seems much more stable and reliable.
ML2013 is unable to open Simulink on any of the workstations. We decided to make the default version of Matlab R2015b (the default of the version of RCG we are using).
I commenced the procedure of the migration, starting with making a tagged commit of the current running simulink models. A local backup was also made, plus we have the usual chiara-based backup so I think we're in good hands.
Currently the branch and tag are protected - once we verify that everything works as expected post migration, I will open it up. I changed the directory structure of the models, need to confirm that the rtcds compilers don't have any hardcoded paths which may break due to my change.
The symlink to Matlab R2013 was deleted and a new symlink to R2015b was made. I activated the license using the Caltech campus license. Now running matlab from shell starts up R2015b . Simulink even works 😲 .
The requirement on the phase noise on the direct backscatter from the OMC back into the SRM is that it be less than @ 100 Hz, for a safety factor (arbitrarily chosen) of 10 (= 20dB below unsqueezed vacuum). Assuming 5 optics between the OMC and SRM which contribute incoherently for a factor of sqrt(5), and assuming a total of 1 ppm of the LO power to be backscattered, we need the suspensions to be moving @ 100 Hz. This seems possible to realize with single stage suspensions - I assume we get f^4 filtering from the pendulum at 100 Hz, and that there is an additional 80 dB attenuation (from the stack) of the assumed 1 micron/rtHz motion at 100 Hz, for an overall 160 dB attenutaiton, yielding 10^-14 m/rtHz at 100 Hz.
This is the same calculation as I had posted a couple of months ago (see elog that this is a reply to), except that Koji pointed out that the LO power is expected to dominate the (carrier) power incident on the OMC cavity(ies). So the more meaningful comparison to make is to have the x-axes of the plots denote the backscatter fraction, rather than the LO power. One subtlety is that because the phase of the scattered field is random, the displacement-noise induced phase noise could show up in the amplitude quadrature. I think that in these quadrature field amplitude units, the RIN and phase noise are directly comparable but I might have missed a factor of 2*pi. But in the worst case, if all the phase noise shows up in the amplitude quadrature, we end up being only ~10dB below unsqueezed vacuum (for 1 ppm backscatter).
For the requirement on the noise in the intensity quadrature - I think this is automatically satisfied because the RIN requirement on the incident LO field is in the mid 10^-9 1/rtHz regime.
Couple IR light into fiber with good MM at EY
At ~1am PDT today, all the MC1 shadow sensor readbacks (fast CDS channels and Slow Acromag channels, latter not shown here) went to negative values. Of course a negative value makes no sense. After ~3 hours, they came back to positive values again. But since then, the shadow sensor RMS noise has been significantly higher in the >20 Hz band, and there are frequent glitches which kick the suspension. The IMC has been having trouble staying locked. I claim that this has to do with the Satellite box.
No action being taken now while I work on the ALS. In the past the problem has fixed itself.
We scoped out the 1Y3 rack this morning to figure out what needs to be done hardware wise. We did not think about how to power the Acromag crate - the LSC rack electronics are all powered by linear supplies and not Sorensens, and the linear supplies are operating at pretty close to their maximum current-drive. The Acromag box draws ~3A of current from the 20 V supply, not sure what the current draw will be from the 15 V supply. Options:
I'm going with option #2 unless anyone has strong objections.
We want to know that we can lock the interferometer with the ALS beat note being generated by beating IR pickoffs (rather than the vertex green transmission). The hope is also to make the ALS system good enough that we can transition the CARM offset directly to 0 after the DRMI is locked with arms held off resonance.
Attachment #1: Shows the layout. The realized MM is ~36 %. c.f. the 85% predicted by a la mode. It is difficult to optimize much more given the tight layout, and the fact that these fast lenses require the beam to be well centered on them. They are reasonably well aligned, but I don't want to futz around with the pointing into the doubling crystal. Consequently, I don't have much control over the pointing.
Attachment #2: Shows pictures of the fiber tips at both ends before/after cleaning. The tips are now much cleaner.
The BeatMouth NF1611 DC monitor reports ~580 mV with only the EY light incident on it. This corresponds to ~60 uW of light making it to the photodiode, which is only 25% of what we send in. This is commensurate with the BS loss + mating sleeve losses.
To find the beat between PSL and EY beams, I had to change the temperature control MEDM slider for the EY laser to -8355 cts (it was 225 cts). Need to check where this lies in the mode-hop scan by actually looking at the X-tal temperature on the front panel of the EY NPRO controller - we want to be at ~39.3 C on the EY X-tal, given the PSL X-tal temp of ~30.61 C. Just checked it, front panel reports 39.2C, so I think we're good.
EY enclosure was closed up and ETMY Oplev was re-enabled after my work. Some cleanup/stray beam dumping remains to be done, I will enlist Chub's help on Monday.
I bench tested the functionality of all the c1iscaux Acromag crate channels. Summary: we are not ready for a Monday install, much debugging remains.
I am leaving the crate powered (by bench supplies) in the office area so I have the option to work remotely on this.
With Chub's help, most of the problems have been resolved. Summary: I judge that we are good to go ahead with an install tomorrow.
Since we don't immediately need the CM board, I say we push ahead with the install - at least that will restore the ability to lock PRMI / DRMI. Then we can debug these issues in situ - I'm certain the issue is related to the EPICS/Modbus setup and not the hardware because I verified the physical channel map using the Acromag windows utility.
Any other ideas? The problem persists and it's annoying that the IMC cannot be locked.
This morning, I wanted to move the existing cables going to the P1 connectors of the iLIGO whitening boards to the P2 connector, to test the modifications made to allow whitening stage switching. Unfortunately, I found that the shrouds werent installed. Where can I find these?
As it turns out, only one extra shroud needed to be installed - I did this and migrated the cables for the 4 whitening boards from the P1 to P2 connectors. So until the new Acromag box is installed, we have no control over the whitening gains (slow channels), but do still have control over the whitening filter enable/disable (controlled by fast BIO). I am thinking about the easiest way to test the latter - I think the ambient PD dark noise level is too low to be seen above ADC noise even with the whitening enabled, and setting up drive signals to individual channels is too painful - maybe with +45dB of whitening gain, the (z,p) whitening filter shape can be seen with just PD/demod chain electroncis noise.
I came aross an interesting suggestion by Yutaro that KAGRA's low-frequency ALS noise could be limited by the fact that the IMC comes between the point where the frequencies of the PSL and AUX lasers are sensed (i.e. the ALS beat note), and the point where we want them to be equal (i.e. the input of the arm cavity). I wanted to see if the same effect could be at play in the 40m ALS system. A first estimate suggests to me that the numbers are definitely in the ballpark. If this is true, we may benefit from lower noise ALS by picking off the PSL beam for the ALS beat note after the IMC.
Even though the KAGRA phase lock scheme is different from the 40m scheme, the algebra holds. I needed an estimate of how much the arm cavity moves, I used data from a POX lock to estimate this. The estimate is probably not very accurate (since the arm cavity length is more stable than the IMC length, and the measured ALS noise, e.g. this elog, is actually better than what this calculation would have me believe), but should be the right order of magnitude. From this crude estimate, it does look like for f<10 Hz, this effect could be significant. I assumed an IMC pole of 3.8 kHz for this calculation.
I've indicated a "target" ALS performance where the ALS noise would be less than the CARM linewidth, which would hopefully make the locking much easier. Seems like realizing this target will be touch-and-go. But if we can implement length feedforward control for the arm cavities using seismometers, the low frequency motion of the optics should go down. It would be interesting to see if the ALS noise gets better at low frequencies with length feedforward engaged.
* Some updates were made to the plot:
Installation: The following equipment were installed in 1Y3, see Attachment #1:
Removal: The following equipment was removed from 1Y3:
I judge that we are good to go ahead with an install tomorrow.
Work done today:
Testing of functionality:
Much testing remains to be done, but I defer further testing till Monday - the main functionality to be verified in the short run is the whitening gain stepping. The strain-relief of cables and general cleanup will be undertaken by Chub. Current state of affairs is in Attachment #3, leaves much to be desired in terms of cleanliness.
I will also setup the autoburt for the new machine on Monday. We will also need to add some channels to C0EDCU.ini if we want to trend them over some years (e.g. RF signal powers for monitoring ERA-5 health).
* c1lsc FE was rebooted using the usual script, and everything seems to be healthy in CDS-land again, see Attachment #4.
Here is what is left to do:
Today I set up the autoburt.req file for the c1iscaux channels, and confirmed that the snapshots are getting recorded. There were a lot of channels in the old autoburt.req file which I thought were un-necessary (and several which no longer exist), so now the only channels that are burt-ed are the whitening gains and states of the AA filters. If someone feels we need more channels to be snapshot recorded, you can add them to the file.
In the old target directory, there were also various versions of a "saverestore.req" file - why do we need this in addition to an autoburt? I guess it is possible they are used by the IFOconfigure scripts to setup some whitening gains etc...
I did some more investigation of what the appropriate cavity geometry would be for the OMC. Unsurprisingly, depending on the incident mode content, the preferred operating point changes. So how do we choose what the "correct" model is? Is it accurate to model the output beam HOM content from NPROs (is this purely determined by the geometry of the lasing cavity?), which we can then propagate through the PMC, IMC, and CARM cavities? This modeling will be written up in the design document shortly.
*Colorbar label errata - instead of 1 W on BS, it should read 1 W on PRM. The heatmaps take a while to generate, so I'll fix that in a bit.
Update 230pm PDT: I realize there are some problems with these plots. The critically coupled f2 sideband getting transmitted through the T=10% SRM should have significantly more power than the transmission through a T=100ppm optic. For similar modulation depth (which we have), I think it is indeed true that there will be x1000 more f2 power than f1 power for both the IFO AS beam and the LO pickoff through the PRC. But if the LO is picked off elsewhere, we have to do the numbers again.
Attachment #1: Two candidate models. The first follows the power law assumption of G1201111, while in the second, I preserved the same scaling, but for the f1 sideband, I set the DC level by assuming a PRG of 45, modulation depth of 0.18, and 100 ppm pickoff from the PRC such that we get 50 mW of carrier light (to act as a local oscillator) for 10 W incident on the back of PRM. Is this a reasonable assumption?
Attachment #2: Heatmaps of the OMC transmission, assuming (i) 0 contrast defect light in the carrier TEM00 mode, (ii) PRG=45 and (iii) 1 W incident on the back of PRM. The color bar limits are preserved for both plots, so the "dark" areas of the plot, which indicate candidate operating points, are darker in the left-hand plot. Obviously, when there is more f1 power incident on the OMC, more of it is transmitted. But my point is that the "best operating point(s)" in both plots are different.
Why is this model refinement necessary? In the aLIGO OMC design, an assumption was made that the light level of the f1 sideband is 1/1000th that of the f2 sideband in the interferometer AS beam. This is justified as the RC lengths are chosen such that the f2 sideband is critically coupled to the AS port, but the f1 is not (it is not quite anti-resonant either). For the BHD application, this assumption is no longer true, as long as the LO beam is picked off after the RF sidebands are applied. There will be significant f1 content as well, and so the mode content of the f1 field is critical in determining the OMC filtering performance.
There were a bunch of useless / degenerate channels added - e.g. whitening gains which are alreay burt-snapshot. Maybe there are many more useless channels being trended, but no need to add more.
Copy-pasting wasn't done correctly - the first 4 added channels were duplicates. There are in fact 5 LO power mons, one for each of the frequencies 11, 33, 55, 110 and 165 MHz.
I cleaned up. Basically only the detect-mon channels, and the ALS channels, are new in the setup now. I will review if any extra channels are required later. While checking that the daqd is happy, I noticed c1lsc FEs are in their stuck state, see Attachment #1. I guess a cable was bumped when the strain relief operation was underway. I'm not attempting a remote resuscitation.
I added the list of new c1iscaux channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini and restarted the framebuilder. Koji had thought some of these channels might have previously existed under slightly different names. However, after looking through C0EDCU.ini and the other _SLOW.ini files, I did not find any candidates for removal. As far as I can tell, all of these channels are being recorded for the first time.
Chub wanted to get the correct part number for the replacement UPS batteries which necessitated opening up the UPS. To be cautious, all the workstations were shutdown at ~9:30am while the unit is pulled out and inspected. While looking at the UPS, we found that the insulation on the main power cord is damaged at both ends. Chub will post photos.
However, despite these precautions, rossa reports some error on boot up (not the same xdisp junk that happened before). pianosa and donatella came back up just fine. It is remotely accessible (ssh-able) though so maybe we can recover it...
please no one touch the UPS: last time it destroyed ROSSA. Please ask Chub to order the replacement batteries so we can do this in a controlled way (fully shutting down ALL workstations first). Last time we wasted 8 hours on ROSSA rebuilding
I installed 6 of these in 1Y2. Three were for PD INTF #1-3, and I used three more for the AS110, REFL11, and REFL33 Demod board FEs, where the strain-reflief of the DC power cables to the Eurocrate was becoming a problem. So now there are only 4 units available as spares.
Once the strain-relieving of the Dsub cabling to 1Y3 is done, we can move ahead with testing. I'd like to put this to bed this week if possible.
Let's not worry about C1LSC until the c1iscaux upgrade is done.
But it still doesn't lock. We notice that the c1lsc machine doesn't work. So we run rebootCILSC.sh.
For some reason, the daqd_fw service was dead on FB. This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 . Simply restarting the fw service does not work, it crashes again after ~20 seconds. The problem may have to do with the indeterminate state of the c1lsc expansion chassis. However, this is not something that can immediately be fixed, as Chub is still working on the wiring there. So in summary, no frame data will be available until we fix this problem (it is still unclear what exactly the problem is). Team WFS can still work by getting online data.
Why were the CDS overview DC indicators not red???
Unrelated to this work: I had to key the c1psl crate to get the IMC autolocker functioning again. However, I found that the key 🔑 turns continuously - as opposed to having two well defined states, ON and OFF. Be careful while handling this.
Came across this while looking up the BIO situation at 1Y2. For reference, the fix Koji mentions can be seen in the attached screenshot (one example, the other BIO cards also have a similar fix). The 16th bit of the BIO is grounded, and some bit-shifting magic is used to implement the desired output.
Yutaro talked about the BIO bug in KAGRA elog. http://klog.icrr.u-tokyo.ac.jp/osl/?r=9536
I think I made the similar change for the 40m model somewhere (don't remember), but be aware of the presense of this bug.
controls@fb1:~ 127$ sudo systemctl status daqd_fw.service
● daqd_fw.service - Advanced LIGO RTS daqd frame writer
Loaded: loaded (/etc/systemd/system/daqd_fw.service; enabled)
Active: active (running) since Tue 2019-09-17 21:32:25 PDT; 17min ago
Main PID: 22040 (daqd_fw)
└─22040 /usr/bin/daqd_fw -c /opt/rtcds/caltech/c1/target/daqd/daqdrc.fw
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer crc thread - label dqprodcrc pid=22108
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] [Tue Sep 17 21:32:31 2019] Producer thread - label dqproddbg pid=22109Producer crc... permitted
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer crc thread put on CPU 0
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer thread priority error Operation not permitted
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer thread put on CPU 0
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer thread - label dqprod pid=22103
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer thread priority error Operation not permitted
Sep 17 21:32:31 fb1 daqd_fw: [Tue Sep 17 21:32:31 2019] Producer thread put on CPU 0
Sep 17 21:32:35 fb1 daqd_fw: [Tue Sep 17 21:32:35 2019] Minute trender made GPS time correction; gps=1252816371; gps%60=51
Sep 17 21:33:31 fb1 daqd_fw: [Tue Sep 17 21:33:31 2019] ->3: clear crc
drwxr-xr-x 2 controls controls 569344 Aug 23 05:17 12465
drwxr-xr-x 2 controls controls 565248 Aug 23 05:41 12466
drwxr-xr-x 2 controls controls 557056 Aug 23 05:53 12505
drwxr-xr-x 2 controls controls 262144 Aug 23 18:40 12506
drwxr-xr-x 2 controls controls 12288 Sep 17 21:54 12528
Unrelated to this work: c1auxey was keyed.
This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 .
INCORRECT INFO IN THIS ELOG HAS BEEN REMOVED. SEE THIS ELOG FOR THE UPDATED INFO.
With the help of a tester board, I verified the mapping between fast BIO DB37 pins, and pins on the IDC50 connectors that are to be broken out to the whitening boards. I will enlist Chub to implement this mapping in hardware later today.
Update 2019 Sep 19 1730: The pin numbers of the IDC 50 connector are all off by 1. i.e. 3-->4 and so on. I will fix this shortly. The problem was because of me looking at the pinout for the wrong gender of IDC50 connectors.
The custom ribbon cables piping the coil driver board outputs to the eLIGO (?) TTs (a.k.a. TT1 and TT2) are damaged. They need to be re-made. I can't find any pin-mapping for them.
While waiting for the LSC photodiode whitening switching cross-connect work to be done, I thought I'd re-align the IFO a bit. However, I was unable to find any beam making it to the REFL/AS ports despite some TT steering. I remembered that Chub had undone the TT connections at 1Y2 as well, and thought I'd check the cabling to make sure all was in order. On going to the rack, however, I found that these connections were damaged at the coil-driver end (see Attachment #1), presumably during the cable extraction. These need to be re-made...😔