40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Nov 11 12:50:10 2009, Jenne, Update, PEM, Seismometer Noise Characteristics HuddleTest_Aug2009data.png
    Reply  Thu Apr 22 20:28:40 2010, Jenne, Update, PEM, Seismometer Noise Characteristics Gur1_Gur2_noise.png
       Reply  Thu May 27 23:59:54 2010, rana, Update, PEM, New Foam Box installed Untitled.png
          Reply  Fri May 28 00:40:53 2010, rana, Update, PEM, DAQ down 
             Reply  Fri May 28 10:44:47 2010, josephb, Update, PEM, DAQ down 
                Reply  Tue Jun 8 18:39:36 2010, rana, Update, PEM, DAQ down 
                   Reply  Tue Jun 8 20:52:25 2010, josephb, Update, PEM, DAQ up (for the moment) 
          Reply  Fri May 28 11:26:35 2010, Jenne, Update, PEM, New Foam Box installed 
          Reply  Tue Jun 1 11:47:14 2010, steve, Update, PEM, lead balls on concrete P1060269.JPG
             Reply  Wed Jun 9 19:47:08 2010, nancy, Update, PEM, lead balls on concrete 
                Reply  Thu Jun 10 07:53:14 2010, Alberto, Update, PEM, LaTeXlabs 
                   Reply  Thu Jun 10 10:58:02 2010, Koji, Update, General, LaTeXlabs 
                      Reply  Thu Jun 10 11:10:21 2010, Alberto, Update, General, LaTeXlabs 
Message ID: 3057     Entry time: Tue Jun 8 20:52:25 2010     In reply to: 3056
Author: josephb 
Type: Update 
Category: PEM 
Subject: DAQ up (for the moment) 

As a test, I did a remote reboot of both Megatron and c1iscex, to make sure there was no code running that might interfere with the dataviewer.  Megatron is behind a firewall, so I don't see how it could be interfering with the frame builder.  c1iscex was only running a test module from earlier today when I was testing the multi-filter matrix part.  No daqd or similar processes were running on this machine either, but it is not behind a firewall at the moment. 

Neither of these seemed to affect the lack of past data.  I note the error message from dataviewer was "read(); errno=9".

Going to the frame builder machine, I ran dmesg.  I get some disturbing messages from May 26th and June 7th. There are 6-7 of these pairs of lines for each of these days, spread over the course of about 30 minutes.

Jun 7 14:05:09 fb ufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full

Jun 7 14:11:14 fb last message repeated 19 times

There's also one:

Jun 7 13:35:14 fb syslogd: /usr/controls/main_daqd.log: No space left on device

I went to /usr/controls/ and looked at the file.  I couldn't read it with less, it errored with Value too large for defined data type.  Turns out the file was 2.3 G.  And had not been updated since June 7th.  There were also a bunch of core dump files from May 25th, and a few more recent.  However the ones from May 25th were somewhat large, half a gig each or so.  I decided to delete the main_daqd.log file as well as the core files.

This seems to have fixed the data history for the moment (at least with one 16k channel I tested quickly). However, I'm now investigating why that log file seems to have filled up, and see if we can prevent this in the future.

Quote:

As before, I am unable to get data from the past. With DTT on Allegra I got data from now, but its unavailable from 1 hour ago. Same problem using mDV on mafalda. I blame Joe again - or the military industrial complex.

 

Quote:

Quote:

 Although trends are available, I am unable to get any full data from in the past (using DTT or DV). I started the FB's daqd process a few times, but no luck. 

I blame Joe's SimPlant monkeying from earlier today for lack of a better candidate. I checked and the frames are actually on the FB disk, so its something else.

 I tried running dataviewer and dtt this morning.  Dataviewer seemed to be working.  I was able to get trends, full data on a 2k channel (seismic channels) and full data on a 16k channel (C1:PEM-AUDIO_MIC1)  This was tried for a period 24 hours a go for a 10 minute stretch.

I also tried dtt and was able to get 2k and 16k channel data, for example C1:PEM-AUDIO_MIC1.  Was this problem fixed by someone last night or did time somehow fix it?

 

ELOG V3.1.3-