This morning I came in the 40m then found
1) c1auxex was throwing out the same errors as recently seen.
2) c1iscex processes had errors which persisted even after the mx stream reset.
1) c1auxex - fixed
Tried telnet c1auxex => rejected by the host
Went down to the south end. Power cycled the target. Came back to the control room.
=> Confirmed the epics read/write is back.
Burtrestored the epics vars for the target to the snapshot on 31th Oct at 5:07.
2) c1iscex - still not fixed
ssh c1iscex
rtcds restart all => c1x01 is still in red.
Followed the procedure on the elog entry 9007. => Still the same error.
At least c1x01 is stalled. Here is the status.
Sync Source is TDS.
C1:DAQ-DC0_C1X01_STATUS is 0x2bad.
C1:DAQ-DC0_C1X01_CRC_SUM stays 0.
The screen shot is attached.
dmesg related to c1x01
controls@c1iscex ~ 0$ dmesg |grep c1x01
[ 32.152010] c1x01: startup time is 1069440223
[ 32.152012] c1x01: cpu clock 3000325
[ 32.152014] c1x01: Epics shmem set at 0xffffc9001489c000
[ 32.152208] c1x01: IPC at 0xffffc90018947000
[ 32.152209] c1x01: Allocated daq shmem; set at 0xffffc9000480c000
[ 32.152210] c1x01: configured to use 4 cards
[ 32.152211] c1x01: Initializing PCI Modules
[ 32.152226] c1x01: ADC card on bus b; device 4 prim b
[ 32.152227] c1x01: adc card on bus b; device 4 prim b
[ 32.154801] c1x01: pci0 = 0xdc300400
[ 32.154837] c1x01: pci2 = 0xdc300000
[ 32.154842] c1x01: ADC I/O address=0xdc300000 0xffffc90003f62000
[ 32.154845] c1x01: BCR = 0x84060
[ 32.154858] c1x01: RAG = 0x117d8
[ 32.154861] c1x01: BCR = 0x84260
[ 32.583220] c1x01: SSC = 0x16
[ 32.583223] c1x01: IDBC = 0x1f
[ 32.583236] c1x01: DAC card on bus 14; device 4 prim 14
[ 32.583237] c1x01: dac card on bus 14; device 4
[ 32.584527] c1x01: pci0 = 0xdc400400
[ 32.584546] c1x01: dac pci2 = 0xdc400000
[ 32.584551] c1x01: DAC I/O address=0xdc400000 0xffffc90003f6a000
[ 32.584555] c1x01: DAC BCR = 0x810
[ 32.584678] c1x01: DAC BCR after init = 0x30080
[ 32.584681] c1x01: DAC CSR = 0xffff
[ 32.584687] c1x01: DAC BOR = 0x3415
[ 32.584693] c1x01: set_8111_prefetch: subsys=0x8114; vendor=0x10e3
[ 32.584722] c1x01: Contec 1616 DIO card on bus 23; device 0
[ 32.593429] c1x01: contec 1616 dio pci2 = 0x4001
[ 32.593430] c1x01: contec 1616 diospace = 0x4000
[ 32.593434] c1x01: contec dio pci2 card number= 0x0
[ 32.593439] c1x01: Contec BO card on bus 18; device 0
[ 32.593447] c1x01: contec dio pci2 = 0x3001
[ 32.593448] c1x01: contec32L diospace = 0x3000
[ 32.593451] c1x01: contec dio pci2 card number= 0x0
[ 32.593456] c1x01: 5565 RFM card on bus 7; device 4
[ 32.597218] Modules linked in: c1x01(+) open_mx mbuf
[ 32.599939] [<ffffffffa002e430>] mapRfm+0x71/0x392 [c1x01]
[ 32.600199] [<ffffffffa002ec91>] mapPciModules+0x540/0x8cf [c1x01]
[ 32.600458] [<ffffffffa002f2c1>] init_module+0x2a1/0x9d6 [c1x01]
[ 32.600717] [<ffffffffa002f020>] ? init_module+0x0/0x9d6 [c1x01]
[ 32.616194] c1x01: RFM address is 0xd8000000
[ 32.616196] c1x01: CSR address is 0xdc000000
[ 32.616206] c1x01: Board id = 0x65
[ 32.616209] c1x01: DMA address is 0xdc000400
[ 32.616213] c1x01: 5565DMA at 0xffffc90003f72400
[ 32.616215] c1x01: 5565 INTCR = 0xf010100
[ 32.616217] c1x01: 5565 INTCR = 0xf000000
[ 32.616218] c1x01: 5565 MODE = 0x43
[ 32.616220] c1x01: 5565 DESC = 0x0
[ 32.616232] c1x01: 5 PCI cards found
[ 32.616233] c1x01: ***************************************************************************
[ 32.616234] c1x01: 1 ADC cards found
[ 32.616235] c1x01: ADC 0 is a GSC_16AI64SSA module
[ 32.616236] c1x01: Channels = 64
[ 32.616236] c1x01: Firmware Rev = 34
[ 32.616238] c1x01: ***************************************************************************
[ 32.616239] c1x01: 1 DAC cards found
[ 32.616239] c1x01: DAC 0 is a GSC_16AO16 module
[ 32.616240] c1x01: Channels = 16
[ 32.616241] c1x01: Filters = None
[ 32.616242] c1x01: Output Type = Differential
[ 32.616242] c1x01: Firmware Rev = 6
[ 32.616244] c1x01: MASTER DAC SLOT 0 1
[ 32.616244] c1x01: ***************************************************************************
[ 32.616246] c1x01: 0 DIO cards found
[ 32.616246] c1x01: ***************************************************************************
[ 32.616248] c1x01: 0 IIRO-8 Isolated DIO cards found
[ 32.616248] c1x01: ***************************************************************************
[ 32.616250] c1x01: 0 IIRO-16 Isolated DIO cards found
[ 32.616250] c1x01: ***************************************************************************
[ 32.616252] c1x01: 1 Contec 32ch PCIe DO cards found
[ 32.616252] c1x01: 1 Contec PCIe DIO1616 cards found
[ 32.616253] c1x01: 0 Contec PCIe DIO6464 cards found
[ 32.616254] c1x01: 2 DO cards found
[ 32.616255] c1x01: TDS controller 0 is at 0
[ 32.616256] c1x01: Total of 4 I/O modules found and mapped
[ 32.616257] c1x01: ***************************************************************************
[ 32.616259] c1x01: 1 RFM cards found
[ 32.616260] c1x01: RFM 0 is a VMIC_5565 module with Node ID 41
[ 32.616261] c1x01: address is 0x18d80000
[ 32.616261] c1x01: ***************************************************************************
[ 32.616262] c1x01: Initializing space for daqLib buffers
[ 32.616263] c1x01: Initializing Network
[ 32.616264] c1x01: Found 1 frameBuilders on network
[ 32.616265] c1x01: Epics burt restore is 0
[ 33.616012] c1x01: Epics burt restore is 0
[ 34.617018] c1x01: Epics burt restore is 0
[ 35.618017] c1x01: Epics burt restore is 0
[ 36.619011] c1x01: Epics burt restore is 0
[ 37.621007] c1x01: Epics burt restore is 0
[ 38.622008] c1x01: Epics burt restore is 0
[ 39.733257] c1x01: Sync source = 4
[ 39.733257] c1x01: Waiting for EPICS BURT Restore = 1
[ 39.793001] c1x01: Waiting for EPICS BURT 0
[ 39.793001] c1x01: BURT Restore Complete
[ 39.793001] c1x01: Found a BQF filter 0
[ 39.793001] c1x01: Found a BQF filter 1
[ 39.793001] c1x01: Initialized servo control parameters.
[ 39.794002] c1x01: DAQ Ex Min/Max = 1 3
[ 39.794002] c1x01: DAQ XEx Min/Max = 3 53
[ 39.794002] c1x01: DAQ Tp Min/Max = 10001 10007
[ 39.794002] c1x01: DAQ XTp Min/Max = 10007 10507
[ 39.794002] c1x01: DIRECT MEMORY MODE of size 64
[ 39.794002] c1x01: daqLib DCU_ID = 19
[ 39.794002] c1x01: Calling feCode() to initialize
[ 39.794002] c1x01: entering the loop
[ 39.794002] c1x01: ADC setup complete
[ 39.794002] c1x01: DAC setup complete
[ 39.794002] c1x01: writing BIO 0
[ 39.814002] c1x01: writing DAC 0
[ 39.814002] c1x01: Triggered the ADC
[ 40.874003] c1x01: timeout 0 1000000
[ 40.874003] c1x01: exiting from fe_code()
|