40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Sat May 16 21:05:24 2015, rana, Update, General, some status 48.png39.png
    Reply  Mon May 18 09:50:00 2015, ericq, Update, General, some status 
       Reply  Mon May 18 11:59:07 2015, rana, Update, General, some status 
          Reply  Tue May 19 16:18:57 2015, ericq, Update, General, crons fixed 
    Reply  Mon May 18 16:28:18 2015, ericq, Update, General, some status 
       Reply  Mon May 18 18:03:12 2015, rana, Update, General, some status CPUtrend.png
          Reply  Tue May 19 00:19:23 2015, rana, Update, General, some status Screen_Shot_2015-05-19_at_12.17.39_AM.png
             Reply  Wed May 20 03:08:27 2015, rana, Update, General, some status 
    Reply  Mon May 18 17:42:14 2015, rana, ericQ, Update, General, some status 
       Reply  Tue May 19 18:55:12 2015, rana, ericQ, Update, General, some status 
          Reply  Wed May 20 11:41:59 2015, ericq, Update, General, some status 
    Reply  Mon Sep 21 00:51:36 2015, rana, Update, General, op340m, autoburt cron =? megatron 
       Reply  Mon Sep 21 11:40:30 2015, ericq, Update, General, Megatron maitenence 
          Reply  Tue Dec 1 17:26:14 2015, Koji, Update, General, Megatron maitenence 
             Reply  Tue Dec 1 20:20:16 2015, Koji, Update, General, Megatron maitenence 
Message ID: 11301     Entry time: Mon May 18 16:28:18 2015     In reply to: 11294     Reply to this: 11305
Author: ericq 
Type: Update 
Category: General 
Subject: some status 
Quote:

4) Noticed that DAQD is restarting once per hour on the hour. Why?

It looks like daqd isn't being restarted, but in fact crashing every hour.

Going into the logs in target/fb/logs/old, it looks like at 10 seconds past the hour, every hour, daqd starts spitting out:

[Mon May 18 12:00:10 2015] main profiler warning: 1 empty blocks in the buffer                                     
[Mon May 18 12:00:11 2015] main profiler warning: 0 empty blocks in the buffer                                     
[Mon May 18 12:00:12 2015] main profiler warning: 0 empty blocks in the buffer                                     
[Mon May 18 12:00:13 2015] main profiler warning: 0 empty blocks in the buffer
...
***CRASH***

An ELOG search on this kind of phrase will get you a lot of talk about FB transfer problems. 

I noticed the framebuilder had 100% usage on its internal, non-RAID, non /frames/, HDD, which hosts the root filesystem (OS files, home directory, diskless boot files, etc), largely due to a ~110GB directory of frames from our first RF lock that had been copied over to the home directory. The HDD only has 135GB capacity. I thought that maybe this was somehow a bottleneck for files moving around, but after deleting the huge directory, daqd still died at 4PM. 

The offsite LDAS rsync happens at ten minutes past the hour, so is unlikely to be the culprit. I don't have any other clues at this point. 

ELOG V3.1.3-