40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed May 25 16:56:44 2022, Paco, Configuration, BHD, IFO recovery - IMC alignment 
    Reply  Wed May 25 17:34:47 2022, yuta, Configuration, BHD, IFO recovery - IFO alignment Screenshot_2022-05-25_17-47-57.png
       Reply  Thu May 26 19:55:43 2022, yuta, Configuration, BHD, Oplevs centered, BHD DCPDs are now online elog_1Y2.JPGelog_BHD.JPGelog_box.JPGScreenshot_2022-05-26_17-37-27_IFOaligned_OplevCentered.pngScreenshot_2022-05-26_20-35-02.png
          Reply  Fri May 27 17:45:53 2022, yuta, Configuration, BHD, BHD camera installed, GRY aligned 
             Reply  Thu Jun 2 20:05:37 2022, yuta, Configuration, PSL, IMC input power recovered to 1W, some alignment works Before.JPGAfter.JPG
                Reply  Fri Jun 3 12:13:58 2022, Paco, Configuration, CDS, Fix RFM channels  SoGreen.png
                   Reply  Fri Jun 3 15:22:51 2022, yuta, Update, LSC, Both arms locked with POY/POX, IR beam centered on TMs with ASS IRBeamsOnTMs.JPGScreenshot_2022-06-03_15-03-51.png
                      Reply  Fri Jun 3 17:42:50 2022, yuta, Update, LSC, MICH locks with AS55_Q Screenshot_2022-06-03_17-41-55.png
                         Reply  Sun Jun 5 19:46:40 2022, Paco, Update, LSC, Fixed IMC Trans sum issue 
                         Reply  Mon Jun 6 13:35:11 2022, Paco, Update, LSC, First calibrated spectra of MICH at AS55 Q Screenshot_2022-06-06_17-30-16_MICHOLTF.pngCalibrated_MICH_ERR.pdf
                            Reply  Fri Jun 17 16:22:21 2022, yuta, Update, LSC, Actuator calibration of BS. ITMX, ITMY, updated MICH displacement spectra from c1cal 6x
                               Reply  Thu Jul 7 18:18:19 2022, yuta, Update, LSC, Actuator calibration of ETMX and ETMX Screenshot_2022-07-05_14-52-01_OLTF.pngScreenshot_2022-07-05_14-54-03_TF.pngScreenshot_2022-07-05_14-56-41_Ratio.png
                                  Reply  Thu Jul 7 18:22:12 2022, yuta, Update, LSC, Actuator calibration of MC2 using Yarm TF.pngMC2.png
                                     Reply  Fri Jul 8 16:18:35 2022, rana, Update, LSC, Actuator calibration of MC2 using Yarm 
                                     Reply  Mon Jul 18 17:07:12 2022, yuta, Update, LSC, x4.12 added to ETMX coil outputs to balance with ETMY 
          Reply  Thu Jun 16 18:23:15 2022, Paco, Configuration, BHD, Recovering LO beam in BHD DCPDs Screenshot_2022-06-16_18-29-14_BHDLObeamISBACK.png
          Reply  Tue Jun 21 14:17:50 2022, yuta, Configuration, BHD, BHD DCPDs re-routed to c1sus2 C1X07ADC1.JPGBHDDCPDs.JPG
             Reply  Tue Jul 19 16:00:34 2022, yuta, Configuration, BHD, Fast channels for BHD DCPDs now available in c1lsc but not in c1hpc Screenshot_2022-07-19_14-26-39_c1hpc.pngScreenshot_2022-07-19_14-24-49_c1lsc.pngScreenshot_2022-07-19_15-51-25_GreenGreen.png
                Reply  Thu Jul 21 21:50:47 2022, Tega, Configuration, BHD, c1sus2 IPC update 
                   Reply  Fri Jul 22 15:05:26 2022, Tega, Configuration, BHD, c1sus2 shared memory and ADC fix Screenshot_2022-07-22_15-04-33_DCPD.pngScreenshot_2022-07-22_15-12-19_models.pngScreenshot_2022-07-22_15-15-11_ERR.pngScreenshot_2022-07-22_15-32-19_GDS.png
                      Reply  Fri Jul 22 17:46:10 2022, yuta, Configuration, BHD, c1sus2 watchdog update and DCPD ERR channels Screenshot_2022-07-22_17-48-25.png
                         Reply  Mon Jul 25 17:58:10 2022, Tega, Configuration, BHD, c1sus2 IPC dolphin issue update Screen_Shot_2022-07-25_at_5.43.28_PM.pngScreen_Shot_2022-07-25_at_5.21.10_PM.pngScreen_Shot_2022-07-25_at_5.30.58_PM.pngScreen_Shot_2022-07-25_at_5.35.19_PM.png
                            Reply  Mon Aug 1 18:42:39 2022, Tega, Configuration, BHD, c1sus2 IPC dolphin issue update c1sus2_dolphin.pngfb1_dxamin_status.pngdolphin_num_mem_init2.png
                               Reply  Tue Aug 2 17:25:18 2022, Tega, Configuration, BHD, c1sus2 dolphin IPC issue solved c1lsc_IPC_status.pngFE_lsmod_dependencies_c1sus2_b4_after_iop_unpdate.png
                      Reply  Mon Jul 25 18:09:41 2022, Tega, Configuration, BHD, BHD Homodyne Phase control MEDM screen Screen_Shot_2022-07-25_at_6.12.08_PM.pngScreen_Shot_2022-07-25_at_6.18.09_PM.png
Message ID: 17052     Entry time: Mon Aug 1 18:42:39 2022     In reply to: 17033     Reply to this: 17054
Author: Tega 
Type: Configuration 
Category: BHD 
Subject: c1sus2 IPC dolphin issue update 

[Yuta, Tega]

We decided to give the dolphin debugging another go. Firstly, we noticed that c1sus2 was no longer recogonising the dolphin card, which can be checked using

lspci | grep Stargen

or looking at the status light on the dolphin card of c1sus2, which was orange for both ports A and B.

We decided to do a hard reboot of c1sus2 and turned off the DAQ chassis for a few minutes, then restared c1sus2. This solved the card recognition problem as well as the 'dis_irm' driver loading issue (I think the driver does not get loaded if the system does not recognise a valid card, as I also saw the missing dis_irm driver module on c1testand). 

Next, we confirmed the status of all dolphin cards on fb1, using

controls@fb1$ /opt/DIS/sbin/dxadmin

It looks like the dolphin card on c1sus2 has now been configured and is availabe to all other nodes. We then restated the all FE machines and models to see if we are in the clear. Unfortunately, we are not so lucky since the problem persisted.

Looking at the output of 'dmesg', we could only identity two notable difference between the operational dolphin cards on c1sus/c1ioo/c1lsc and c1sus2, namely: the card number being equal to zero and the memory addresses which are also zero, see image below.

Anyways, at least we can now eliminate driver issues and would move on to debugging the models next.

Attachment 1: c1sus2_dolphin.png  330 kB  Uploaded Mon Aug 1 20:48:25 2022  | Hide | Hide all
c1sus2_dolphin.png
Attachment 2: fb1_dxamin_status.png  73 kB  Uploaded Mon Aug 1 20:50:16 2022  | Hide | Hide all
fb1_dxamin_status.png
Attachment 3: dolphin_num_mem_init2.png  363 kB  Uploaded Mon Aug 1 20:51:38 2022  | Hide | Hide all
dolphin_num_mem_init2.png
ELOG V3.1.3-