40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Dec 14 19:41:00 2017, gautam, Update, CDS, CDS recovery, NFS woes CDS_14Dec2017.pngCDS_errors.png
    Reply  Fri Dec 15 00:26:40 2017, johannes, Update, CDS, Re: CDS recovery, NFS woes 
    Reply  Fri Dec 15 01:53:37 2017, jamie, Update, CDS, CDS recovery, NFS woes 
       Reply  Fri Dec 15 11:19:11 2017, gautam, Update, CDS, CDS recovery, NFS woes NFS.pngMCautolocker.png
Message ID: 13481     Entry time: Fri Dec 15 11:19:11 2017     In reply to: 13480
Author: gautam 
Type: Update 
Category: CDS 
Subject: CDS recovery, NFS woes 

Looking at the dmesg on c1iscex for example, at least part of the problem seems to be associated with FB1 (192.168.113.201, see Attachment #1). The "server" can be unresponsive for O(100) seconds, which is consistent with the duration for which we see the MEDM status lights go blank, and the EPICS records get frozen. Note that the error timestamped ~4000 was from last night, which means there have been at least 2 more instances of this kind of freeze-up overnight.

I don't know if this is symptomatic of some more widespread problem with the 40m networking infrastructure. In any case, all the CDS overview screen lights were green today morning, and MC autolocker seems to have worked fine overnight.

I have also updated the wiki page with the updated daqd restart commands.

Unrelated to this work - Koji fixed up the MC overview screen such that the MC autolocker button is now visible again. The problem seems to do with me migrating some of the c1ioo EPICS channels from the slow machine to the fast system, as a result of which the EPICS variable type changed from "ENUM" to something that was not "ENUM". In any case, the button exists now, and the MC autolocker blinky light is responsive to its state.

Quote:

I don't think the problem is fb1.  The fb1 NFS is mostly only used during front end boot.  It's the rtcds mount that's the one that sees all the action, which is being served from chiara.

 

Attachment 1: NFS.png  90 kB  | Hide | Hide all
NFS.png
Attachment 2: MCautolocker.png  27 kB  | Hide | Hide all
MCautolocker.png
ELOG V3.1.3-