40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Feb 21 17:29:57 2011, rana, Summary, IOO, Myterious data loss: FB needs investigation Untitled.png
    Reply  Wed Feb 23 10:37:02 2011, josephb, Summary, IOO, Myterious data loss: FB needs investigation 
Message ID: 4343     Entry time: Wed Feb 23 10:37:02 2011     In reply to: 4333
Author: josephb 
Type: Summary 
Category: IOO 
Subject: Myterious data loss: FB needs investigation 

Friday: 

In addition to the other fixes, Alex rebuilt the daqd process. I failed to elog this. When he rebuilt it, he needed change the symmerticom gps offset in the daqdrc file (located in /opt/rtcds/caltech/c1/target/fb). 

On Friday night, Kiwamu contacted me and let me know the frame builder had core dumped after a seg fault.  I had him temporarily disable the c1ass process (the only thing we changed that day), and then replaced Alex's rebuilt daqd code with the original daqd code and restarted it.  However, I did not change the symmetricom offset at this point.  Finally, I restarted the NDS process.  At that point testpoints and  trends seemed to be working.

Sunday:

The daqd process was restarted sometime on Sunday night (by Valera i believe).  Apparently this restart finally had the symmetricom gps offset kick in (perhaps because it was the first restart after the NDS was restarted?).  So data was being written to a future gps time.

Monday:

Kiwamu had problems with testpoints and trends and contacted me.  I tracked down the gps offset and fixed it, but the original daqd process only started once successfully, after that is was segfault, core dump non-stop. I tried Alex's rebuilt daqd (along with putting the gps offset to the correct value for it), and it worked.  Test points, trends, excitations were checked at the point and found working.

I still do not understand the underlying causes of all these segmentation faults with both the old and new daqd codes.  Alex has suggested some new open mx drivers be installed today.

Quote:

Looks like there was a mysterious loss of data overnight; since there's nothing in the elog I assume that its some kind of terrorism. I'm going to call Rolf to see if he can come in and work all night to help diagnose the issue.


 

ELOG V3.1.3-