40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 3861     Entry time: Thu Nov 4 15:27:33 2010
Author: josephb, yuta 
Type: Update 
Category: CDS 
Subject: c1ioo test points not working 

Problem:

We can't access any of the c1ioo computer testpoints.  Dataviewer and DTT both fail to read any data from them.

According to diag -l (when run on Rosalba in /opt/apps), the testpoints are not being set.  Also at some point during the day when debugging, we also somehow messed up all the front end connections to the framebuilder.

Errors reported by dataviewer:

Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
read(); errno=0
Server error 6532: invalid channel name
Server error 16080: unknown error
datasrv: DataWriteRealtime failed: daq_send: Illegal seek

Error reported by daqd.log:

[Thu Nov  4 13:29:35 2010] About to request `C1:IOO-MC1_PIT_IN1' 10022
on node 34
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10022; tp[1]=32531

[Thu Nov  4 13:29:35 2010] About to request `C1:SUS-MC2_SUSPOS_IN1'
10094 on node 36
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10094; tp[1]=32531

[Thu Nov  4 13:29:38 2010] ETIMEDOUT: test point `C1:IOO-MC1_PIT_IN1'
(tp_num=10022) was not set by the test point manager; request failed
[Thu Nov  4 13:29:38 2010] About to clear `C1:IOO-MC1_PIT_IN1' 10022 on node 34

Attempted Fixes:

Remove all daq related files: /opt/rtcds/caltech/c1/target/gds/param/tpchn_c1ioo.par and /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini.  Rebuilt the front end code.

Double checked ethernet connection between c1ioo and the daq router and fb. 

Confirmed open mx was running on c1ioo. Confirmed awgtpman was running (although at one point I did find duplicate awgtpman running for c1x03, the IOP associated with c1ioo).

Rebooted the c1ioo machine. Confirmed all necessary codes came back.

Restarted the daqd process several times on the fb machine.

Current Status:

Framebuilder is running, and c1sus testpoints are available.  c1ioo test points are not.  Waiting to hear back from Alex on possible ideas.

ELOG V3.1.3-