40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Dec 20 21:29:41 2018, Jon, Omnistructure, Upgrade, Vacuum Controls Switchover Completed 40m_vacuum_acromag_channels.pdf
    Reply  Fri Dec 21 11:13:13 2018, gautam, Omnistructure, VAC, N2 line valved off VacGauges.pngScreenshot_from_2018-12-21_13-02-06.png
       Reply  Fri Dec 21 12:57:10 2018, Koji, Omnistructure, VAC, N2 line valved off 
          Reply  Thu Jan 3 15:08:37 2019, gautam, Omnistructure, VAC, Vac status unknown Screenshot_from_2019-01-03_15-19-51.pngScreenshot_from_2019-01-03_15-14-14.png997B13A9-CAAF-409C-A6C2-00414D30A141.jpeg
             Reply  Fri Jan 4 17:43:24 2019, gautam, Update, CDS, Timing issues 
                Reply  Wed Jan 9 11:33:35 2019, gautam, Update, CDS, Timing issues still persist gpstimeSync.pngScreenshot_from_2019-01-09_17-56-58.png
          Reply  Fri Jan 4 10:25:19 2019, Jon, Omnistructure, VAC, N2 line valved off 
    Reply  Thu Mar 21 18:36:59 2019, Jon, Omnistructure, Upgrade, Vacuum Controls Switchover Completed 40m_Vacuum_Acromag_Channels_20190321.pdf
Message ID: 14386     Entry time: Fri Jan 4 17:43:24 2019     In reply to: 14380     Reply to this: 14392
Author: gautam 
Type: Update 
Category: CDS 
Subject: Timing issues 

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

  • Now, old frame data is no longer being overwritten
  • For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

  • The usual FE/FB power cycling did not fix the problem.
  • The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
  • This was off by 2 years.
  • The way this worked up till now was by adding a fixed offset to this time.
    • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
    • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
    • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

  1. First, we tried changing the offset in the daqdrc.fw file only.
    • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
    • This did not fix the problem.
  2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
    • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
    • Added the following lines:
      /* 2018 had no leap seconds or leap days, so adjust for that */
             pHardware->gpsOffset += 31536000;
    • re-built and installed the modified symmetricom driver.
    • Checked that cat /proc/gps now yields the correct time.
    • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
    • With these steps, the frames were being written to /frames with the correct timestamp.
  3. Next, we checked the timing on the FEs
    • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
    • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
    • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
    • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
    • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Quote:
 
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?
ELOG V3.1.3-