40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Apr 21 10:35:23 2022, Koji, Update, CDS, DAQ seemed down 
    Reply  Mon Apr 25 17:24:06 2022, Anchal, Update, CDS, DAQ still down 
Message ID: 16811     Entry time: Mon Apr 25 17:24:06 2022     In reply to: 16793
Author: Anchal 
Type: Update 
Category: CDS 
Subject: DAQ still down 

I investigated this issue today. At first, it seemed that only new suspension testpoints are inaccessible. I was able to use diaggui for a measurement on MC2. The DAQ network cable between 1X4 and 1Y1 was tied and is very taught now (we should relieve this as soon as possible, best solution is to lay down a longer cable over the bridge). My hypothesis is that the DAQ network might have broken while tying this cable and it probably did not come back since then.

The simplest solution would have been to restart c1su2 models. As I restarted those models though, I found that c1lsc and c1sus models failed. This is very unusual as c1su2 models are independent and share nothing with the other vertex models. So I had to restart all the FE computers eventually. But this did not solve the issue. Worse, now the DAQ isn't working for the vertex machiens as well.

Next step was to try restarting fb1 itself. We switched off all the FE computers, restarted fb1, stopped daqd_* processes, reloaded gpstime module, restarted open-mx, mx, nds and daqd_* process. But the mx.service failed to load with following error message:

● mx.service - LSB: starts MX driver for Myrinet card
   Loaded: loaded (/etc/init.d/mx)
   Active: failed (Result: exit-code) since Mon 2022-04-25 17:18:02 PDT; 1s ago
  Process: 4261 ExecStart=/etc/init.d/mx start (code=exited, status=1/FAILURE)

Apr 25 17:18:02 fb1 mx[4261]: Loading mx driver
Apr 25 17:18:02 fb1 mx[4261]: insmod: ERROR: could not insert module /opt/mx/sbin/mx_mcp.ko: File exists
Apr 25 17:18:02 fb1 mx[4261]: insmod: ERROR: could not insert module /opt/mx/sbin/mx_driver.ko: File exists
Apr 25 17:18:02 fb1 systemd[1]: mx.service: control process exited, code=exited status=1
Apr 25 17:18:02 fb1 systemd[1]: Failed to start LSB: starts MX driver for Myrinet card.
Apr 25 17:18:02 fb1 systemd[1]: Unit mx.service entered failed state.

(Ignore the timestamp above, I ran the command again to capture the error message.)

However, I was able to start all the FE models without any errors and daqd processes are also all running without showing any errors. Everything is green in CDS screen with no error messages. But the only thing still wrong is mx.service which is not running.

From my limited knowledge and experience, mx.service is a one-time script that mounts mx devices in /dev and loads the mx driver. I tried running the script /opt/mx/sbin/mx_start_stop :

controls@fb1:/opt/mx/sbin 1$ sudo ./mx_start_stop start
Loading mx driver
insmod: ERROR: could not insert module /opt/mx/sbin/mx_mcp.ko: File exists
insmod: ERROR: could not insert module /opt/mx/sbin/mx_driver.ko: File exists

This gave the same error. On searching little bit online, "insmod: ERROR; cound not insert module" error comes up when the kernel version of the driver doesnot match the Linux kernel (whatever that means!). Such deep issues should not appear out of nowhere in a previosuly perfectly runnig system. I'll check around more what changed in fb1, network cables etc.

ELOG V3.1.3-