40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m elog  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Mon Aug 6 19:37:50 2012, Jamie, Update, CDS, daqd and CDS network problems today 
    Reply  Mon Aug 6 19:54:53 2012, Jamie, Update, CDS, daqd and CDS network problems today 
       Reply  Mon Aug 6 20:08:45 2012, Jamie, Update, CDS, daqd and CDS network problems today 
          Reply  Mon Aug 6 20:22:50 2012, Jamie, Update, CDS, daqd segfaulting after five minutes 
             Reply  Tue Aug 7 11:46:24 2012, Jamie, Update, CDS, Alex working on daqd 
                Reply  Tue Aug 7 14:17:07 2012, Jamie, Update, CDS, daqd running again; related to c1sup issue 
                   Reply  Tue Aug 7 14:34:01 2012, Jamie, Update, CDS, jk. daqd still segfaulting 
                      Reply  Tue Aug 7 15:04:23 2012, Jamie, Update, CDS, daqd problem was root-owned files and directories 
Message ID: 7096     Entry time: Mon Aug 6 20:22:50 2012     In reply to: 7095     Reply to this: 7101
Author: Jamie 
Type: Update 
Category: CDS 
Subject: daqd segfaulting after five minutes 

I tried running daqd manually, and sure enough it segfaults after about five minutes (see log below).  I've uncommented it from /etc/inittab on fb and I'm leaving it off for now until we can figure out what's going on.

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ /opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc
263596
MX endpoint opened
startup file interpreter thread tid=139790943115536
calling yyparse(5, 6)
[Mon Aug  6 20:15:27 2012] ->5: #set avoid_reconnect
[Mon Aug  6 20:15:27 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 20:15:27 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 20:15:27 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 20:15:27 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 20:15:27 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 20:15:27 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 20:15:27 2012] ->5: set debug=0
[Mon Aug  6 20:15:27 2012] ->5: set log=2
[Mon Aug  6 20:15:27 2012] ->5: set zero_bad_data=0
[Mon Aug  6 20:15:27 2012] ->5: set dcu_status_check=9
[Mon Aug  6 20:15:27 2012] ->5: set controller_dcu=33
[Mon Aug  6 20:15:27 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 20:15:30 2012] finished configuring data channels
[Mon Aug  6 20:15:30 2012] ->5: configure channels begin end
GDS server NODE=19 HOST=c1iscex DCUID=19
GDS server NODE=20 HOST=c1sus DCUID=20
GDS server NODE=21 HOST=c1sus DCUID=21
GDS server NODE=22 HOST=c1lsc DCUID=22
GDS server NODE=25 HOST=c1iscex DCUID=61
GDS server NODE=28 HOST=c1ioo DCUID=28
GDS server NODE=33 HOST=c1ioo DCUID=33
GDS server NODE=34 HOST=c1ioo DCUID=34
GDS server NODE=36 HOST=c1sus DCUID=36
GDS server NODE=38 HOST=c1sus DCUID=38
GDS server NODE=39 HOST=c1sus DCUID=39
GDS server NODE=40 HOST=c1lsc DCUID=40
GDS server NODE=42 HOST=c1lsc DCUID=42
GDS server NODE=45 HOST=c1iscex DCUID=45
GDS server NODE=46 HOST=c1iscey DCUID=46
GDS server NODE=47 HOST=c1iscey DCUID=47
GDS server NODE=48 HOST=c1lsc DCUID=48
GDS server NODE=50 HOST=c1lsc DCUID=50
GDS server NODE=51 HOST=c1ioo DCUID=51
GDS server NODE=60 HOST=c1lsc DCUID=60
GDS server NODE=61 HOST=c1iscex DCUID=61
GDS server NODE=62 HOST=c1sus DCUID=62
Unable to find GDS node 90 system c1x00 in INI files
GDS server NODE=91 HOST=c1lsc DCUID=60
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
TP: node = 19, host = c1iscex, dup = 0, prog = 0x31002013, vers = 1
Initialized TP interface node=19, host=c1iscex
TP: node = 20, host = c1sus, dup = 0, prog = 0x31002014, vers = 1
Initialized TP interface node=20, host=c1sus
TP: node = 21, host = c1sus, dup = 0, prog = 0x31002015, vers = 1
Initialized TP interface node=21, host=c1sus
TP: node = 22, host = c1lsc, dup = 0, prog = 0x31002016, vers = 1
Initialized TP interface node=22, host=c1lsc
TP: node = 25, host = c1iscex, dup = 0, prog = 0x31002019, vers = 1
Initialized TP interface node=25, host=c1iscex
TP: node = 28, host = c1ioo, dup = 0, prog = 0x3100201c, vers = 1
Initialized TP interface node=28, host=c1ioo
TP: node = 33, host = c1ioo, dup = 0, prog = 0x31002021, vers = 1
Initialized TP interface node=33, host=c1ioo
TP: node = 34, host = c1ioo, dup = 0, prog = 0x31002022, vers = 1
Initialized TP interface node=34, host=c1ioo
TP: node = 36, host = c1sus, dup = 0, prog = 0x31002024, vers = 1
Initialized TP interface node=36, host=c1sus
TP: node = 38, host = c1sus, dup = 0, prog = 0x31002026, vers = 1
Initialized TP interface node=38, host=c1sus
TP: node = 39, host = c1sus, dup = 0, prog = 0x31002027, vers = 1
Initialized TP interface node=39, host=c1sus
TP: node = 40, host = c1lsc, dup = 0, prog = 0x31002028, vers = 1
Initialized TP interface node=40, host=c1lsc
TP: node = 42, host = c1lsc, dup = 0, prog = 0x3100202a, vers = 1
Initialized TP interface node=42, host=c1lsc
TP: node = 45, host = c1iscex, dup = 0, prog = 0x3100202d, vers = 1
Initialized TP interface node=45, host=c1iscex
TP: node = 46, host = c1iscey, dup = 0, prog = 0x3100202e, vers = 1
Initialized TP interface node=46, host=c1iscey
TP: node = 47, host = c1iscey, dup = 0, prog = 0x3100202f, vers = 1
Initialized TP interface node=47, host=c1iscey
TP: node = 48, host = c1lsc, dup = 0, prog = 0x31002030, vers = 1
Initialized TP interface node=48, host=c1lsc
TP: node = 50, host = c1lsc, dup = 0, prog = 0x31002032, vers = 1
Initialized TP interface node=50, host=c1lsc
TP: node = 51, host = c1ioo, dup = 0, prog = 0x31002033, vers = 1
Initialized TP interface node=51, host=c1ioo
TP: node = 60, host = c1lsc, dup = 0, prog = 0x3100203c, vers = 1
Initialized TP interface node=60, host=c1lsc
TP: node = 61, host = c1iscex, dup = 0, prog = 0x3100203d, vers = 1
Initialized TP interface node=61, host=c1iscex
TP: node = 62, host = c1sus, dup = 0, prog = 0x3100203e, vers = 1
Initialized TP interface node=62, host=c1sus
TP: node = 91, host = c1lsc, dup = 0, prog = 0x3100205b, vers = 1
Initialized TP interface node=91, host=c1lsc
[Mon Aug  6 20:15:30 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 20:15:30 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 20:15:30 2012] ->5: set detector_name="CIT"
[Mon Aug  6 20:15:30 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 20:15:30 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 20:15:30 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 20:15:30 2012] ->5: set detector_elevation=.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 20:15:30 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 20:15:30 2012] ->5: set num_dirs = 10
[Mon Aug  6 20:15:30 2012] ->5: set frames_per_dir=225
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 20:15:30 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 20:15:30 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 20:15:30 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: start main 10
Allocated move buffer size 11616356 bytes
[Mon Aug  6 20:15:32 2012] main started
[Mon Aug  6 20:15:32 2012] ->5: start profiler
[Mon Aug  6 20:15:32 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 20:15:32 2012] frame saver started
[Mon Aug  6 20:15:32 2012] ->5: start frame-saver
[Mon Aug  6 20:15:33 2012] ->5: sync frame-saver
[Mon Aug  6 20:15:33 2012] ->5: start trender
[Mon Aug  6 20:15:33 2012] trender started
[Mon Aug  6 20:15:33 2012] trend frame saver started
[Mon Aug  6 20:15:33 2012] ->5: start trend-frame-saver
[Mon Aug  6 20:15:34 2012] ->5: sync trend-frame-saver
[Mon Aug  6 20:15:34 2012] minute trend frame saver started
[Mon Aug  6 20:15:34 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 20:15:34 2012] Done creating ADC structures
[Mon Aug  6 20:15:35 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 20:15:35 2012] raw minute trend frame saver started
[Mon Aug  6 20:15:35 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 20:15:35 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 20:15:35 2012] ->5: #sleep 5
[Mon Aug  6 20:15:35 2012] producer started
[Mon Aug  6 20:15:35 2012] ->5: start producer
[Mon Aug  6 20:15:35 2012] ->5: start epics dcu
[Mon Aug  6 20:15:35 2012] MX receiver thread started
[Mon Aug  6 20:15:35 2012] edcu started
[Mon Aug  6 20:15:35 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 20:15:35 2012] epics server started
[Mon Aug  6 20:15:35 2012] ->5: start listener 8087
[Mon Aug  6 20:15:35 2012] ->5: start listener 8088 1
[Mon Aug  6 20:15:35 2012] ->5: sleep 60
Creating C1:DAQ-DC0_PEM_SLOW_STATUS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_CPS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_SUM
Creating C1:DAQ-DC0_C1X01_STATUS
Creating C1:DAQ-DC0_C1X01_CRC_CPS
Creating C1:DAQ-DC0_C1X01_CRC_SUM
Creating C1:DAQ-DC0_C1X02_STATUS
Creating C1:DAQ-DC0_C1X02_CRC_CPS
Creating C1:DAQ-DC0_C1X02_CRC_SUM
Creating C1:DAQ-DC0_C1SUS_STATUS
Creating C1:DAQ-DC0_C1SUS_CRC_CPS
Creating C1:DAQ-DC0_C1SUS_CRC_SUM
Creating C1:DAQ-DC0_C1OAF_STATUS
Creating C1:DAQ-DC0_C1OAF_CRC_CPS
Creating C1:DAQ-DC0_C1OAF_CRC_SUM
Creating C1:DAQ-DC0_C1ALS_STATUS
Creating C1:DAQ-DC0_C1ALS_CRC_CPS
Creating C1:DAQ-DC0_C1ALS_CRC_SUM
Creating C1:DAQ-DC0_C1X03_STATUS
Creating C1:DAQ-DC0_C1X03_CRC_CPS
Creating C1:DAQ-DC0_C1X03_CRC_SUM
Creating C1:DAQ-DC0_C1IOO_STATUS
Creating C1:DAQ-DC0_C1IOO_CRC_CPS
Creating C1:DAQ-DC0_C1IOO_CRC_SUM
Creating C1:DAQ-DC0_C1MCS_STATUS
Creating C1:DAQ-DC0_C1MCS_CRC_CPS
Creating C1:DAQ-DC0_C1MCS_CRC_SUM
Creating C1:DAQ-DC0_C1RFM_STATUS
Creating C1:DAQ-DC0_C1RFM_CRC_CPS
Creating C1:DAQ-DC0_C1RFM_CRC_SUM
Creating C1:DAQ-DC0_C1PEM_STATUS
Creating C1:DAQ-DC0_C1PEM_CRC_CPS
Creating C1:DAQ-DC0_C1PEM_CRC_SUM
Creating C1:DAQ-DC0_C1X04_STATUS
Creating C1:DAQ-DC0_C1X04_CRC_CPS
Creating C1:DAQ-DC0_C1X04_CRC_SUM
Creating C1:DAQ-DC0_C1LSC_STATUS
Creating C1:DAQ-DC0_C1LSC_CRC_CPS
Creating C1:DAQ-DC0_C1LSC_CRC_SUM
Creating C1:DAQ-DC0_C1SCX_STATUS
Creating C1:DAQ-DC0_C1SCX_CRC_CPS
Creating C1:DAQ-DC0_C1SCX_CRC_SUM
Creating C1:DAQ-DC0_C1X05_STATUS
Creating C1:DAQ-DC0_C1X05_CRC_CPS
Creating C1:DAQ-DC0_C1X05_CRC_SUM
Creating C1:DAQ-DC0_C1SCY_STATUS
Creating C1:DAQ-DC0_C1SCY_CRC_CPS
Creating C1:DAQ-DC0_C1SCY_CRC_SUM
Creating C1:DAQ-DC0_C1ASS_STATUS
Creating C1:DAQ-DC0_C1ASS_CRC_CPS
Creating C1:DAQ-DC0_C1ASS_CRC_SUM
Creating C1:DAQ-DC0_C1CAL_STATUS
Creating C1:DAQ-DC0_C1CAL_CRC_CPS
Creating C1:DAQ-DC0_C1CAL_CRC_SUM
Creating C1:DAQ-DC0_C1MCC_STATUS
Creating C1:DAQ-DC0_C1MCC_CRC_CPS
Creating C1:DAQ-DC0_C1MCC_CRC_SUM
Creating C1:DAQ-DC0_C1MCP_STATUS
Creating C1:DAQ-DC0_C1MCP_CRC_CPS
Creating C1:DAQ-DC0_C1MCP_CRC_SUM
Creating C1:DAQ-DC0_C1LSP_STATUS
Creating C1:DAQ-DC0_C1LSP_CRC_CPS
Creating C1:DAQ-DC0_C1LSP_CRC_SUM
Creating C1:DAQ-DC0_C1SPX_STATUS
Creating C1:DAQ-DC0_C1SPX_CRC_CPS
Creating C1:DAQ-DC0_C1SPX_CRC_SUM
Creating C1:DAQ-DC0_C1SUP_STATUS
Creating C1:DAQ-DC0_C1SUP_CRC_CPS
Creating C1:DAQ-DC0_C1SUP_CRC_SUM
[Mon Aug  6 20:15:35 2012] Epics server started
[Mon Aug  6 20:15:35 2012] EDCU has 2553 channels configured; first=0

Symmetricom status: LOCKED
Starting at gps 1028344552 prev_gps 1028344552 frac 312500000 f 314094022
[Mon Aug  6 20:15:38 2012] Minute trender made GPS time correction; gps=1028344552; gps%60=52
Segmentation fault (core dumped)
ELOG V3.1.3-