As a test, I did a remote reboot of both Megatron and c1iscex, to make sure there was no code running that might interfere with the dataviewer. Megatron is behind a firewall, so I don't see how it could be interfering with the frame builder. c1iscex was only running a test module from earlier today when I was testing the multi-filter matrix part. No daqd or similar processes were running on this machine either, but it is not behind a firewall at the moment.
Neither of these seemed to affect the lack of past data. I note the error message from dataviewer was "read(); errno=9".
Going to the frame builder machine, I ran dmesg. I get some disturbing messages from May 26th and June 7th. There are 6-7 of these pairs of lines for each of these days, spread over the course of about 30 minutes.
Jun 7 14:05:09 fb ufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Jun 7 14:11:14 fb last message repeated 19 times
There's also one:
Jun 7 13:35:14 fb syslogd: /usr/controls/main_daqd.log: No space left on device
I went to /usr/controls/ and looked at the file. I couldn't read it with less, it errored with Value too large for defined data type. Turns out the file was 2.3 G. And had not been updated since June 7th. There were also a bunch of core dump files from May 25th, and a few more recent. However the ones from May 25th were somewhat large, half a gig each or so. I decided to delete the main_daqd.log file as well as the core files.
This seems to have fixed the data history for the moment (at least with one 16k channel I tested quickly). However, I'm now investigating why that log file seems to have filled up, and see if we can prevent this in the future.
Quote: |
As before, I am unable to get data from the past. With DTT on Allegra I got data from now, but its unavailable from 1 hour ago. Same problem using mDV on mafalda. I blame Joe again - or the military industrial complex.
|
Quote:
|
Quote: |
Although trends are available, I am unable to get any full data from in the past (using DTT or DV). I started the FB's daqd process a few times, but no luck. 
I blame Joe's SimPlant monkeying from earlier today for lack of a better candidate. I checked and the frames are actually on the FB disk, so its something else.
|
I tried running dataviewer and dtt this morning. Dataviewer seemed to be working. I was able to get trends, full data on a 2k channel (seismic channels) and full data on a 16k channel (C1:PEM-AUDIO_MIC1) This was tried for a period 24 hours a go for a 10 minute stretch.
I also tried dtt and was able to get 2k and 16k channel data, for example C1:PEM-AUDIO_MIC1. Was this problem fixed by someone last night or did time somehow fix it?
|
|
|