40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Tue Jun 30 11:33:00 2015, Jamie, Summary, CDS, prepping for CDS upgrade 
    Reply  Wed Jul 1 19:16:21 2015, Jamie, Summary, CDS, CDS upgrade in progress 
       Reply  Tue Jul 7 18:27:54 2015, Jamie, Summary, CDS, CDS upgrade: progress! 2.9-RTS-OK.pngC1X04_GDS_TP.png
          Reply  Wed Jul 8 20:37:02 2015, Jamie, Summary, CDS, CDS upgrade: one step forward, two steps back 
             Reply  Wed Jul 8 21:02:02 2015, Jamie, Summary, CDS, CDS upgrade: another step forward, so we're back to where we started (plus a bit?) 
                Reply  Thu Jul 9 13:26:47 2015, Jamie, Summary, CDS, CDS upgrade: new mx 1.2.16 installed 
                   Reply  Thu Jul 9 16:50:13 2015, Jamie, Summary, CDS, CDS upgrade: if all else fails try throwing metal at the problem 
                      Reply  Mon Jul 13 01:11:14 2015, Jamie, Summary, CDS, CDS upgrade: current assessment 
                         Reply  Mon Jul 13 18:12:50 2015, Jamie, Summary, CDS, CDS upgrade: left running in semi-stable configuration 
                            Reply  Tue Jul 14 09:08:37 2015, Jamie, Summary, CDS, CDS upgrade: left running in semi-stable configuration 
                               Reply  Tue Jul 14 10:28:02 2015, ericq, Summary, CDS, CDS upgrade: left running in semi-stable configuration 
                                  Reply  Tue Jul 14 11:57:27 2015, jamie, Summary, CDS, CDS upgrade: left running in semi-stable configuration 
                            Reply  Tue Jul 14 16:51:01 2015, Jamie, Summary, CDS, CDS upgrade: problem is not disk access 
                               Reply  Wed Jul 15 13:19:14 2015, Jamie, Summary, CDS, CDS upgrade: reducing mx end-points as last ditch effort 
                                  Reply  Wed Jul 15 18:19:12 2015, Jamie, Summary, CDS, CDS upgrade: tentative stabilty? 
                                     Reply  Sat Jul 18 15:37:19 2015, Jamie, Summary, CDS, CDS upgrade: current status cds-good.pngsus-damped.png
Message ID: 11409     Entry time: Tue Jul 14 11:57:27 2015     In reply to: 11408
Author: jamie 
Type: Summary 
Category: CDS 
Subject: CDS upgrade: left running in semi-stable configuration 
Quote:

There remains a pattern to some of the restarts, the following times are all reported as restart times. (There are others in between, however.)

daqd: Tue Jul 14 00:02:48 PDT 2015
daqd: Tue Jul 14 01:02:32 PDT 2015
daqd: Tue Jul 14 03:02:33 PDT 2015
daqd: Tue Jul 14 05:02:46 PDT 2015
daqd: Tue Jul 14 06:01:57 PDT 2015
daqd: Tue Jul 14 07:02:19 PDT 2015
daqd: Tue Jul 14 08:02:44 PDT 2015
daqd: Tue Jul 14 09:02:24 PDT 2015
daqd: Tue Jul 14 10:02:03 PDT 2015

Before the upgrade, we suffered from hourly crashes too:

daqd_start Sun Jun 21 00:01:06 PDT 2015
daqd_start Sun Jun 21 01:03:47 PDT 2015
daqd_start Sun Jun 21 02:04:04 PDT 2015
daqd_start Sun Jun 21 03:04:35 PDT 2015
daqd_start Sun Jun 21 04:04:04 PDT 2015
daqd_start Sun Jun 21 05:03:45 PDT 2015
daqd_start Sun Jun 21 06:02:43 PDT 2015
daqd_start Sun Jun 21 07:04:42 PDT 2015
daqd_start Sun Jun 21 08:04:34 PDT 2015
daqd_start Sun Jun 21 09:03:30 PDT 2015
daqd_start Sun Jun 21 10:04:11 PDT 2015

So, this isn't neccesarily new behavior, just something that remains unfixed. 

That's interesting, that we're still seeing those hourly crashes.

We're not writing out the full set of channels, though, and we're getting more failures than just those at the hour, so we're still suffering.

ELOG V3.1.3-