First, we re-balanced the ITMX OSEMs so that they would damp at around half-a-shadow.
Then, we set up a clean camera inside ITMX chamber looking at the ITMX optic. Then, using the live feed we aligned the AUX beam from the ETMX station using M1 and M2. The camera was great to help us align the beam properly close to the ITMX center. It wasn't very long until we could see a green beam on the IR card, but we didn't really see any flashing, so this may just be the bare transmission away from XARM resonance (Attachment #1).
Ian checked the reflection from ITMX using the IR card with holes, and he pretty much only saw one beam spot, so we turned to look for a beam scattering on the vacuum tube but didn't really see anything. This could mean that we were hitting the ETMX again, or missing slightly, or missing completely. We tried scanning the ITMX pitch and yaw using the bias (alignment) sliders, and with the illuminators off, try seeing some scattered green beam on the ETMX. We can't really see anything yet, but we will keep trying. If there are any tips on our method, it would be great to know them.
I'm not sure why but the PIT and YAW offset values of +2725 and -2341 were not sufficient for the reflected OPLEV beam to reach the OPLEV QPD. I had to change the C1:SUS-ETMY_PIT_OFFSET to 5641 and C1:SUS-ETMY_YAW_OFFSET to -4820 to come back to center of the OPLEV QPD. We aligned the Oplevs to center before venting, so hopefully this is our desired ETMY position.
On another note, the issue of ETMY unable to damp was simple. The alignment offsets were on to begin with with values above 1000. This meant that whenever we enabled coil output, ETMY would necessarily get a kick. All I had to do was keep alignment offsets off before starting the damping and slowly increase the alignment offsets to desired position.
- This modification allowed me to align the oplev spot to the center of the QPD. C1:SUS-ETMY_PIT_OFFSET and C1:SUS-ETMY_YAW_OFFSE are +2725 (8%FS) and -2341 (7%FS), respectively.
C1:SUS-ETMY_YAW_OFFSE are +
Continuing with the previous alignment that we stoped on friday, we re set up my heavily cleaned iPhone on FaceTime. The Phone alowed us to see the laser on the ITMX and center it on that optic.
After Ian updated the cts2um filters for OSEM, shouldn't the damping gains be increased back by factor of 10 to previos values? Was the damping gain for SIDE ever changed? we found it at 250.
Can you explain why gain_offset filter was required and why this wasn't done for the side coil?
I updated the gain of the ct2um filter on the OSEMS for ETMY and decreased their gain by a factor of 9 from 0.36 to 0.04.
I added a filter called "gain_offset" to all the coils except for side and added a gain of 0.48.
together these should negate the added gain from the electronics replacement of the ETMY
The point of changing the gains was to return the system to its origional state. ie I wanted the over all gain of the physical components to be the same as when we started. From the CDS side of things nothing else should be changed. The damping filters should remain at their origional values. The cts2um filter was changed to counteract a change in the electronics (replacing them). These changes should cancel eachother out. As for the side control, on 3/4/22 koji reduced the output resistors for the 4 face OSEMs but did not change the the SD one. there fore the SD did not need the same adjustment as the others.
Here is an early sketch of the MC table.
I have made an editable draw.io diagram of the planned simplified BHD setup on the ITMY table (see attached). 10 pts = 1 inch.
This is very sketchy but easily adjustable since we are removing everything but the ITMY Oplev from that table.
We are supposed to have BS Oplev Beams. We don't like the shallow angle reflections (i.e. AOI>45deg).
The laser is too big but I suspect the other components are too small. So it'd be check the actual size of the components including the optical mounts that are missing on the figure so far.
Possibility to swap BS and ITMX tables:
BS table, which Tega said MC table, is 2ft x 4ft. The ITMX table is 3ft x 5ft and only the central 2ft x 4ft area is used. The area around the BS table is the narrowest for the east arm. We need at least (2+delta) ft of the hallway width so that we can move the instrument. I'm not yet sure if the ITMX table can be placed there without precise investigation.
I have updated the BS table using feedback from Koji and Paco and the attached pdf document is the latest iteration.
I begin modeling the initial BHD setup using Finesse. I started with copying C1_w_BHD.kat from the 40m/bhd repo and making changes to reflect the current BHD setup:
1. OMCs were removed.
2. Only 1 PD per BHD port was left.
3. Transmission of PR2 was changed to 2.2%. The PRG was calculated to be ~15.5.
4. Actual RoCs of new optics were dialed in (Yesterday me and Paco went into the cleanroom to measure the RoCs and they seem to match the datasheets).
Here's a table comparing the old (design?) RoCs with the new RoCs:
The changes looked quite alarming, especially for LO4 and AS3, so I wrote a script to calculate the mode matching between the LO and AS beams called AS_LO_ModeMatching.ipynb and pushed it into the repo. In the notebook a bright AS beam is created by creating a small asymmetry between the arms of ~ 0.003 degrees (~10pm). Amplitude detectors were put at the input ports of the BHD BS to calculate the fields in the AS and LO beams. In particular TEM00, TEM02 and TEM20 were measured for each beam.
The calculation shows that with the old RoCs the mode matching between the LO and AS beams is 99% while for the new RoCs it is ~ 50%.
To access the board remotely through the 40m lab ethernet port, use
ssh -N -L localhost:1137:localhost:9090 xilinx@<ip_address>
Then in the browser go to
Other SSH commands using different ports or without the -N -L seemed to fail to open Jupyter. This way has been successful thereafter.
Since last week I've worked with tommy on getting the RFSoC 2x2 board to get some TFs from simple minicircuits type filters. The first thing I did was set up the board (which is in the office area) for remote access. I hooked up the TCP/IP port to a wall ethernet socket (LIGO-04) and the caltech network assiggned some IP address to our box. I guess eventually we can put this behind the lab network for internal use only.
After fiddling around with the tone-generators and spectrum analyzer tools in loopback configuration (DAC --> ADC direct connection), we noticed that lower frequency (~ 1 MHz) signals were hardly making it out/back into the board... so we looked at some of the schematics found here and saw that both RF data converters (ADC & DAC) interfaces are AC coupled through a BALUN network in the 10 - 8000 MHz band (see Attachment #1). This is in principle not great news if we want to get this board ready for audio-band DSP.
We decided that while Tommy works on measuring TFs for SHP-200 all the way up to ~ 2 GHz (which is possible with the board as is) I will design and put together an analog modulation/demodulation frontend so we can upconvert all our "slow" signals < 1MHz for fast, wideband DSP. and demodulate them back into the audio band. The BALUN network is pictured in Attachment #2 on the board, I'm afraid it's not very simple to bypass without damaging the PCB or causing some other unwanted effect on the high-speed DSP.
In the "Tommy" sub folder, I created a new notebook called "SimpleToneGenerator". This tunes the DAC and ADC mixers to a single frequency and reads off the Time Series and Fourier components. We can alos easily check the demodulation scheme and implement butterworth filters to check their function.
In this file (under Tommy), we have a notebook which runs through a spectrum of frequencies and determines the gain response of the attached filter. Below we have the output of a high pass filter. We use IQ demodulation to change IQ componets to DC. Then using a butterworth filter, we read out the DC components and determine the gain's magnitude and phase. However, the phase seems very noisy. This is because the oscillators in the different tiles are independent and a random phase is introduced by changing the mixer frequency in individual tiles. To resolve this we need Multi Tile Synchronization or "MTS".
Original Pynq Support Forum Query: https://discuss.pynq.io/t/rfsoc-2x2-phase-measurement/3892
We also have the code to fit a resposne function using IIRregular, but this is not as useful without proper phase data.
Ok, it turns out these optics were purchased on purpose, as this elog shows. Jon considered building a mode-matching telescope with stock optics as an initial step before purchasing the custom optics (referred to as "design" optics in my elog).
I dialed in the new distances between the optics into the .kat file as described in this elog and pushed the changes to the repo. With the new distances, I got mode-matching of 87% for the full IFO and 89% for FPMI so there's probably no need to worry as the mode-matching with these optics was already designed.
Seems like it should be possible to just remove the transformer (aka as a BALUN ... BALanced, UNbalanced), or replace it with a lower frequency part. Its just a usual mini-circuits part. Maybe you can ask Chris Stoughton about this and ask Tommy to checkout some of the RFSoC user forums for how to go to DC.
[Paco, JC, Ian, Jordan, Chub]
Checking in the morning, I walked over to the Nitrogen tanks to check the levels. Noticed one tank was empty, so I swapped it out. Chub came over to check the levels and to take note of how many tanks were left available for usage (None). Chub continued to put in a work order for a set of full Nitrogen tanks. We should be set on Nitrogen until Thursday this week (4/14/22).
As for C1VAC, this morning, Paco and I attempted to open the PSL shutter, but the interlock system was tripped so we didn't get any light into the IFO. We traced the issue down to C1VAC being unresponsive. We discussed this may have interlocked as a result of the Nitrogen tanks running out, but we do not believe this was the issue since we would have recieved an email. We tried troubleshooting as much as possible avoiding a reboot, but were unable to solve the issue. In response, we ran the idea of a reboot across Jordan and Ian, where everyone was in agreement, and fixed the system. Restarting c1vac seems to have closed V4, but this didn't cause any issues with the current state of the vacuum system.
After opening the PSL shutter again, we see the laser down the IFO, so we resume alignment work
Today, Tega and I would like to vent the pump spool an dinstall the new FRG-400 Agilent Pressure Gauges (per elog 15703). The attached picture shows the volume needed to be vented highlighted in red, and the gauges that need to be replaced/removed (purple dot next to the name).
The vent plan is as follows:
Shut down TP2
Install new gauges
Will add to post with updates post vent.
[Jordan, JC, Tega]
We have installed all the FRGs and updated the VAC medm screens to display their sensor readings. The replacement map is CC# -> FRG#, where # in [1..4] and PRP1 -> FRG5. We now need to clean up the C1VAC python code so that it is not overloaded with non-function gauges (CC1,CC2,CC3,CC4,PRP1). Also, need to remove the connection cables for the old replaced gauges.
Tested the Nikon batteries for the camera. they are supposed to be 7V batteries but they don't hold a charge. I confirmed this with multi-meter after charging for days. Ordered new ones Nikon EN-EL9
[Ian, Paco, JC]
There is a strange smell in the 40m. It smells like a chemically burning smell maybe like a shorted component. I went around with the IR camera to see if anything was unusually hot but I didn't see anything. The smell seems to be concentrated at the vertex and down the y-arm
I believe that the Nikon has an exposure problem and that's why we bought the Canon.
Prior to venting the RGA volume on Tuesday (4/12/2022) I took an RGA scan of the volume to be vented (RGA+TP1 volume+Manual Gate Valve) to see if there was a difference after replacing the manual gate valve. Attached is the plot from 4/12/22, and an overlay plot to complare 4/12/22 to 12/10/2021, when the same volume was scanned with the old (defective) manual gate valve.
There is a significant drop in the ratio O2 compared the the nitrogen peak and reduced Argon (AMU 40) which indicates there is no longer a large air leak.
12/10/21 N2/O2 ratio ~ 4 (Air 78%N2 / 21%O2)
4/12/22 N2/O2 ratio ~ 10
There is one significant (above noise level) peak above AMU 46, which is at AMU 58. This could possibly be acetone (AMU 43 and 58) but overall the new RGA Volume scans look significantly better after the manual gate valve replacement. Well done!
Came this morning, opened the PSL and there was not even a beam on the MC REFL.
Looking at the big monitor it seems like the WFS signals went through the roof during the "auto-alignment" night session.
I restored the MC alignment from before the misalignment happen and wait for the SUS to damp. Once the RMS values went below 200 I enabled the watchdog and the coil outputs.
I opened the PSL shutter and the IMC locked immediately. I turned on the WFS servo and the MC REFL DC went down to 0.3. I run the WFS relief script.
I cleaned up around the 40 m lab. All the Laser Safety Glasses have been picked up and placed on the rack at the entrance.
Some miscellaneous BNC Connector cables have been arranged and organized along the wall parallel to the Y-Tunnel.
Nitrogen tanks have been swapped out. Current tank is at 1200 psi and the other is at 1850 psi.
The tool box has been organized with each tool in its specified area.
I'm planning on simulating the BHD readout noise in a manner very similar to the ALS noise model using Simulink. I've made a sketch of the model for the longitudinal DOFs (attached). A model for ASC will be similar but with more measurement devices (OpLevs, QPDs, WFSs).
I'm not pretending to simulate everything in this diagram on the first go, it is just a sketch of the big picture.
We set up POX11 beam path from ITMX chamber to the ITMX in-air table. To do this, we first identified the POX reflection on the ITMX chamber, and then steered the POXM1 (in the BSC) by hand until we cleared the viewport. We also checked that the POX beam is centered on POXM1.
We then decided to slide the LO1 YAW to clear the LO beam path, which was otherwise clipping on the PR2 SOS. The slider (DAC limited) range of -25000 counts was barely enough to clear the SOS comfortably and avoid hitting the POXM1. The LO beam is now hitting LO2 mirror, so LO alignment can proceed from BSC and ITMY chamber.
Finally, we aligned the input ITMX Oplev beam to ITMXOL2, then ITMX, then to ITMXOL2, and finally into the ITMX in-air optical table. We took some photos of the Oplev beam (see Attachments) to note their position.
By the end of in-vacuum work there was still some flashing in the arm cavities, but fine alignment is required.
After closing the ITMX Chamber, and BSC, we moved on to center the ITMXOL beam. We accomplished this by using two mirrors instead of one as was previously the case. This relaxed the angle of incidence a little, but we had to change the path and the position of the QPD. The QPD sum reads ~ 6600 counts versus the ~ >8000 counts it read right before the vent. One attempt at closing the OL loop, and the ITMX starting oscillating in YAW (PIT was ok), so we realized that maybe we flipped the order in which the OL1 / OL2 mirror were arranged and so the YAW loop needed to flip its sign. Indeed after changing the C1:SUS-ITMX_OL_YAW_GAIN from -6.0 to 6.0 the OL_YAW loop is stable.
Yesterday, I tried tuning the PR2 damping gains by increasing the gain until the damping gave the ~5 oscillations (by watching the damped motion using StripTool, and keeping an eye on the PD var). I noticed that often when I changed the gain, some OSEM sensors shifted (gained an offset!!) and the PD var values changed, typically increasing at higher damping gains. I reverted the changes until the PD var looked "normal" again (~ 2 mV) but it is hard to imagine that the damping filters can have such a "DC" effect, given the shape comprises single zero at 0 Hz (and pole at 30 Hz).
Here are a few options for replacement BALUNs from Mini Circuits and specs:
Current. TCM1-83X+, 10-8000 MHz, 50 Ohms, Impedance Ratio 1, Configuration K
1. Z7550-..., DC-2500 MHz (some DC-2300), 50/75 Ohms, Impedance Ratio 1.5, Configuration Q. There are various types of the Z7550 which have different connectors (SMA and BNCs). These have much larger dimensions than the TCM1-83X. Can handle up to 5A DC current with matching loss 0.6 dB.
2. SFMP-5075+, DC-2500 MHz, 50/75 Ohms, Impedance Ratio 1.5, Configuration D. This is an SMA connected BALUN. It can handle 350mA, has a matching loss 0.4 dB, and has 1W power handling.
To give more alignment ranges for the SUS alignment, we started updating the output resistors of the BHD SUS coil drivers.
As Paco has already started working on LO1 alignment, we urgently updated the output Rs for LO1 coil drivers.
LO1 Coil Driver 1 now has R=100 // 1.2k ~ 92Ohm for CH1/2/3, and LO1 Coil Driver 2 has the same mod only for CH3. JC has taken the photos and will upload/update an elog/DCC.
We are still working on the update for the SR2, LO2, AS1, and AS4 coil drivers. They are spread over the workbench right now. Please leave them as they're for a while.
JC is going to continue to work on them tomorrow, and then we'll bring them back to the rack.
It seemed that it comes from the servo oscillation. This does not happen when the output limitters were set to be 100-ish. But even so the gains looked quite low.
I turned on the Cheby rool-offs for all the DOFs, and this allowed me to increase the damping gain A LOT.
The gains were 2~5 but now they are now 20-25 for the face OSEMs and 150 for SD.
The attached is the example of the damping when all the damping loops are on.
I think we need to tune the servo loops carefully for all the SUSs by actually looking at the openloop transfer functions rather than a personal feeling. => To Do
Yesterday, when I worked on the damping servo, I found that any of the daqvtools (ndscope, dtt, dataviewer,...) is not available. We may need to restart the fb and rt machines.
It was brought to attention during yesterday's meeting that the pressures in the vacuum system were not equivalent althought the valve were open. So this morning, Jordan and I reviewed the pressure gauges P3 and P4. We attempted to recalibrate, but the gauges were unresponsive. Following this, we proceeded to connect new gauges on the outside to test for a calibration. The two gauges successfully calibrated at atmosperic pressure. We then proceeded to remove the old gauges and install the new ones.
Quick report: JC has done all the mods for the coil driver circuit in the morning and we worked on the reinstallation of them in the afternoon.
I'll check the damping loops / sus servo settings. JC is going to make an ELOG entry and DCC updates for more precise record of the mods.
git repo - https://git.ligo.org/40m/vac
Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.
[Paco, JC, Yuta]
We aligned the BS oplev using the new BSOL mirror pair. The main change is now the AOI of the oplev on the BS is quite normal. The output beam in the in-air table was quite large (diverging?) so we had to place a short FL lens in front of the QPD.
Separately, I added the LO1 YAW offset of ~ -2500 counts (before the coil driver changes it was -24500 counts) and saw LO beam hitting LO2. This means the alignment of the LO beam can move downstream.
BS, ITMX and ETMX were aligned to get flashing in the X arm.
I aligned the POX beam on the ITMX table using a mixture of the old POP and POX optics. The beam was stirred to the POX11 RFPD. We measure the DC power using a scope but we see nothing. We went and saw that the POX11 cable was not connected to RF rack so we connected it along with some other RFPD cables.
We return but there is still no DC. We ndscope C1:LSC-POX11_I_ERR_DQ C1:LSC-POX11_Q_ERR_DQ and maximize the signal (attachment). The readout is very weak though. It should be as strong as POY which we already observed to have good SNR.
We also noticed that the one of the beam dumps for the POX RFPD is not glued and easily falls down.
[Tega, Yuta, Paco]
We tried aligning the LO and AS beams on to the BHD beamsplitter. During the alignment process, we noticed that the damping loop for AS1 was not working. Paco drew our attention to the fact that the UR OSEM signal was alway close to zero, so we checked to ensure that the magnet was still within the OSEM recess and it looks OK. Next we checked the electrical connection at the interface between the copper OSEM cables to the blue in-vacuum flat cable and this too looks alright also. Since the AS1 coil driver was recently modified, it is possible we might find the problem there, so I'll ask Koji about this.
So Koji clarified that the coil driver board and SATAMP boards are different so we should connect this issue to the coil driver board.
Tega and Paco reported that the UR OSEM of AS1 lost the response.
- I have checked the LED MON (left) of the satellite amp for AS1. CH1/2/3 had 5V -> This indicates that the OSEM LEDs are (most likely) functioning.
- Then I went to the ITMY flange and connected the OSEM emulator instead of the Dsub25 cable. The attachment shows that the UR OSEM LED/PD worked fine with the OSEM emulator. WIth the vacuum flange connected it lost the response.
This indicates that the AS1 UR OSEM problem is localized in the chamber. Please check if the DSUB pins are touching the table or something else.
Coil Drivers LO2, SR2, AS4, and AS1 have been updated a reinstalled into the system.
LO2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100008
LO2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100530
SR2 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100614
SR2 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100615
AS1 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100610
AS1 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100611
AS4 Coil Driver 1 (UL/LL/UR)now has R=100 // 1.2k ~ 92Ohm for CH1/2/3 Unit: S2100612
AS4 Coil Driver 2 (LR/SD)now has R=100 // 1.2k ~ 92Ohm for CH3 Unit: S2100613
[JC, Tega, Ian, Paco]
We found that the UR cable was clamped to the table by one of the ITMY OPLEV steering mirror mounts that was recently installed. After freeing the cable, the UR signal is now active again.
Nice. Please put this information on the DCC pages of the coil driver units. You'll find links to all the units in this document tree LIGO-E2100447. For each page, click on "Change Metadata" from the left panel and add the change made to the resistor (including the resistor name on PCB, previous and new value), and add a link to your previous elog post which has more details like photos, to "Notes and Changes", and upload an updated version of the circuit schematic by creating an annotation in the previous circuit schematic pdf. Every unit that has a serial number in the lab has a DCC page (if not, we should create one) where we should track all such hard changes.
If someone gets time, let's put in all the cable posts and clean up our cable routing on the tables.
While the pumpspool is vented, I thought it would be a convenient time to change out the tip seal on the TP3 forepump. This one had not been changed since 2018, so as preventative maintence I had JC remove the pump and begin cleaning/installing the new tip seal.
Unfortunately the tip seal broke, but I have ordered another. We should have this pump ready to go late next week. If one is needed sooner, there is a spare IDP 7 pump we can install as the TP3 forepump.
We recieved an overlay from Chris Stoughton which he used for a ZCU11 board. The overlay is supposed to be compatible with the RFSoC 2x2 and help enable the Multi-Tile Synchronization (MTS) we need. He also provides a .py with the necessary low level connection to the board and its memory along with a few sample notebooks.
Progress So Far:
Coming in this morning, I checked on the Nitrogen tanks to check the level. One of the tanks were empty, so I went ahead and swapped it out. One tank is at 946 PSI, the other is at 2573 PSI. I checked for leaks and found none.
For your (and mine) info:
N2 pressure can be monitored on the 40m summary page: https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20220425/vacuum/
(you need to hit "today" to go to the current status)
[Anchal, Tega, JC]
We installed cable posts in ITMY, BS, and ITMX chambers for all the new suspensions. Now, there is no point where the OSEM connections are hanging freely.
In BS chamber, we installed one post for LO2 near the north edge of the table and another post for PR3 on the Western edge with the blue cable running around the table on the floor.
In ITMY chamber, we installed the cable post in between AS1 and AS4 with the blue cables running around the table on the floor. This is to ensure the useful part of the table remains empty for future and none of the OSEM cables are taught in air.
I investigated this issue today. At first, it seemed that only new suspension testpoints are inaccessible. I was able to use diaggui for a measurement on MC2. The DAQ network cable between 1X4 and 1Y1 was tied and is very taught now (we should relieve this as soon as possible, best solution is to lay down a longer cable over the bridge). My hypothesis is that the DAQ network might have broken while tying this cable and it probably did not come back since then.
The simplest solution would have been to restart c1su2 models. As I restarted those models though, I found that c1lsc and c1sus models failed. This is very unusual as c1su2 models are independent and share nothing with the other vertex models. So I had to restart all the FE computers eventually. But this did not solve the issue. Worse, now the DAQ isn't working for the vertex machiens as well.
Next step was to try restarting fb1 itself. We switched off all the FE computers, restarted fb1, stopped daqd_* processes, reloaded gpstime module, restarted open-mx, mx, nds and daqd_* process. But the mx.service failed to load with following error message:
● mx.service - LSB: starts MX driver for Myrinet card
Loaded: loaded (/etc/init.d/mx)
Active: failed (Result: exit-code) since Mon 2022-04-25 17:18:02 PDT; 1s ago
Process: 4261 ExecStart=/etc/init.d/mx start (code=exited, status=1/FAILURE)
Apr 25 17:18:02 fb1 mx: Loading mx driver
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_mcp.ko: File exists
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_driver.ko: File exists
Apr 25 17:18:02 fb1 systemd: mx.service: control process exited, code=exited status=1
Apr 25 17:18:02 fb1 systemd: Failed to start LSB: starts MX driver for Myrinet card.
Apr 25 17:18:02 fb1 systemd: Unit mx.service entered failed state.
(Ignore the timestamp above, I ran the command again to capture the error message.)
However, I was able to start all the FE models without any errors and daqd processes are also all running without showing any errors. Everything is green in CDS screen with no error messages. But the only thing still wrong is mx.service which is not running.
From my limited knowledge and experience, mx.service is a one-time script that mounts mx devices in /dev and loads the mx driver. I tried running the script /opt/mx/sbin/mx_start_stop :
This gave the same error. On searching little bit online, "insmod: ERROR; cound not insert module" error comes up when the kernel version of the driver doesnot match the Linux kernel (whatever that means!). Such deep issues should not appear out of nowhere in a previosuly perfectly runnig system. I'll check around more what changed in fb1, network cables etc.