40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 45 of 357  Not logged in ELOG logo
ID Date Author Type Category Subjectup Text Attachments
  2911   Tue May 11 16:38:16 2010 josephb,rana,rolfUpdateCDSCDS questions and thoughts1) What is c1asc doing?  What is ascaux
used for?  What are the cables labeled
"C1:ASC_QPD" in the 1X2 rack really
  
  13669   Thu Mar 8 01:10:22 2018 gautamUpdateGeneralCDS recovery after work at LSC rackThis required multiple hard reboots, but
seems like all the RT models are back for
now. The only indicator I can't explain
 CDS-recovery.png 
  13670   Thu Mar 8 14:41:25 2018 gautamUpdateGeneralCDS recovery after work at LSC rackAs I had found before,
restarting the c1oaf
 CDS-recovery.pngc1sus_timing.pngc1sus_crashed.png 
  13672   Thu Mar 8 18:15:42 2018 gautamUpdateGeneralCDS recovery after work at LSC rackI was forced into a simultaneous power-cycle
rebooting of the three vertex FEs just now.
I took the opportunity to completely disconnect
  
  13477   Thu Dec 14 19:41:00 2017 gautamUpdateCDSCDS recovery, NFS woes[Koji, Jamie(remote), gautam]

Summary:
The CDS system seems to be back up and functioning.
 CDS_14Dec2017.pngCDS_errors.png 
  13480   Fri Dec 15 01:53:37 2017 jamieUpdateCDSCDS recovery, NFS woes


Quote:


  
  13481   Fri Dec 15 11:19:11 2017 gautamUpdateCDSCDS recovery, NFS woesLooking at the dmesg on c1iscex for example,
at least part of the problem seems to be
associated with FB1 (192.168.113.201, see
 NFS.pngMCautolocker.png 
  9449   Fri Dec 6 21:38:27 2013 KojiUpdateLSCCDS related activities for LSCI worked on the CDS related stuffs for
LSC yesterday and today.

 LSC1.pngLSC2.png 
  3034   Wed Jun 2 11:25:16 2010 josephb,alexUpdateCDSCDS saga (aka the bad code saga)Alex updated the awg.par file to handle
all the testpoints.  Basically its very
similar to the testpoint.par, but the prognum
  
  14149   Thu Aug 9 12:31:13 2018 gautamUpdateCDSCDS status updateThe model seems to have run without issues
overnight. Not completely related, but the
MC1 shadow sensor signals also don't
  
  14166   Wed Aug 15 21:27:47 2018 gautamUpdateCDSCDS status updateStarting c1cal
now, let's see if the other c1lsc FE
models are affected at all... Moreover, since
  
  14192   Tue Sep 4 10:14:11 2018 gautamUpdateCDSCDS status updatec1lsc crashed again. I've contacted
Rolf/JHanks for help since I'm out of
ideas on what can be done to fix this
  
  14193   Wed Sep 5 10:59:23 2018 wgautamUpdateCDSCDS status updateRolf came by today morning. For now, we've
restarted the FE machine and the expansion
chassis (note that the correct order in which
  
  9077   Wed Aug 28 00:41:23 2013 JenneUpdateCDSCDS svn commits not happeningsvn status update. asx, als and ioo
were found not committed. Not sure about who modified
  
  9079   Wed Aug 28 05:21:58 2013 manasaUpdateCDSCDS svn commits not happeningI am responsible for missed
svn commits with als and asx. I have committed them.
  
  13166   Fri Aug 4 09:07:28 2017 ranaUpdateCDSCDS system essentially NOT fully recoveredTried getting   
  13153   Mon Jul 31 18:44:40 2017 JamieUpdateCDSCDS system essentially fully recoveredThe CDS system is mostly fully recovered
at this point.  The mx_streams are all
flowing from all front ends, and from all
 02.png 
  11687   Tue Oct 13 17:04:54 2015 ericqUpdateCDSCDS thingsAfter some discussion at last week's
40m meeting, I increased the frequency of
daqd trying to write out minute trends from
  
  3829   Sat Oct 30 05:27:53 2010 yutaSummaryCDSCDS time delay measurementMotivation:
  We want to know the time delay
of CDS in the IOP scheme.
 delaysetup.pngCDSdelay.png 
  3830   Sat Oct 30 14:35:43 2010 KojiSummaryCDSCDS time delay measurementUnsatisfactory.
Neglecting the digital anti-imaging
 CDS_system_investigation_090323.pdf 
  3838   Mon Nov 1 15:47:15 2010 yutaSummaryCDSCDS time delay measurementBackground:
  I measured CDS time delay last
week, but because of my lack of understanding
 downupsampling.pngCDSdelay2.png 
  3839   Mon Nov 1 16:43:24 2010 KojiSummaryCDSCDS time delay measurementUm, Beautiful.   
  3961   Sat Nov 20 03:37:11 2010 yutaSummaryCDSCDS time delay measurement - the ripple(Koji, Joe, Yuta)

Motivation:
 c1tst16Kdelay.png 
  4302   Tue Feb 15 15:06:25 2011 josephbUpdateCDSCDS todo list for tomorrow morningCurrently, there is a test directory called
/opt/rtcds/caltech/c1/new_core where we have
the latest svn checkout.  Tomorrow (after
  
  3036   Wed Jun 2 17:34:33 2010 josephb, alex, valeraUpdateCDSCDS updatesFrom what I understand, Alex rewrote portions
of the framebuilder and testpoint codes and
then recompiled them in order to get more
  
  4445   Mon Mar 28 15:18:04 2011 josephbUpdateCDSCDS updates on FridayLast Friday, we discovered a bug in the
RCG where the delay part was not actually
delaying.  We reported this to Alex
  
  17074   Wed Aug 10 20:51:14 2022 TegaUpdateComputersCDS upgrade Front-end machine setupHere is a summary of what needs doing following
the chat with Jamie today.

 
 IMG_20220810_171002928.jpgIMG_20220810_171019633.jpg 
  6540   Tue Apr 17 11:05:04 2012 JamieUpdateCDSCDS upgrade in progressI am continuing to attempt to upgrade the
CDS system to RTS 2.5.  Systems will
continue to be up and down for the rest of
  
  6541   Tue Apr 17 19:03:09 2012 JamieUpdateCDSCDS upgrade in progressUpgrade progresses, but not complete. 
There are some relatively minor issues, and
one potentially big issue.
  
  11390   Wed Jul 1 19:16:21 2015 JamieSummaryCDSCDS upgrade in progressThe CDS upgrade is now underway

Here's
what's happened so far:


Installed and linked in
  
  6552   Fri Apr 20 19:54:57 2012 JamieUpdateCDSCDS upgrade problemsI ran into a couple of snags today.
A big one is that the framebuilder
daqd started going haywire when I told it
  
  6546   Wed Apr 18 19:59:48 2012 JamieUpdateCDSCDS upgrade successThe upgrade is nearly complete:

    new daqd code is running
on fb
    the fe/daqd timing issue
  
  16881   Fri May 27 17:46:48 2022 PacoSummaryComputersCDS upgrade visit, downfall and rise of c1lsc models[Paco, Anchal-remote, Yuta, JC]

Sometime around noon today, right
after cds upgrade planning tour, c1lsc FE
  
  11397   Wed Jul 8 21:02:02 2015 JamieSummaryCDSCDS upgrade: another step forward, so we're back to where we started (plus a bit?)Koji did a bit of googling to determine
that 'Wrong Network' status message
could be explained by the fb myrinet 
  
  11402   Mon Jul 13 01:11:14 2015 JamieSummaryCDSCDS upgrade: current assessmentdaqd is still behaving unstably. 
It's still unclear what the issue is.

The current failures look like
  
  11427   Sat Jul 18 15:37:19 2015 JamieSummaryCDSCDS upgrade: current statusSo it appears we have found a semi-stable
configuration for the DAQ system post upgrade:

 cds-good.pngsus-damped.png 
  11400   Thu Jul 9 16:50:13 2015 JamieSummaryCDSCDS upgrade: if all else fails try throwing metal at the problemI roped Rolf into coming over and adding
his eyes to the problem.  After much
discussion we couldn't come up with any
  
  11404   Mon Jul 13 18:12:50 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configurationI have been watching daqd all day and I
don't feel particularly closer to understanding
what the issues are.  However, things
  
  11406   Tue Jul 14 09:08:37 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configurationOvernight daqd restarted itself only about
twice an hour, which is an improvement:


controls@fb /opt/rtcds/caltech/c1/target/fb
  
  11408   Tue Jul 14 10:28:02 2015 ericqSummaryCDSCDS upgrade: left running in semi-stable configurationThere remains a pattern to some of the
restarts, the following times are all
reported as restart times. (There are others
  
  11409   Tue Jul 14 11:57:27 2015 jamieSummaryCDSCDS upgrade: left running in semi-stable configuration


  
  11398   Thu Jul 9 13:26:47 2015 JamieSummaryCDSCDS upgrade: new mx 1.2.16 installedI rebuilt/installed mx 1.2.16 to use "ether-mode",
instead of the default MX-10G:


controls@fb
  
  11396   Wed Jul 8 20:37:02 2015 JamieSummaryCDSCDS upgrade: one step forward, two steps backAfter determining yesterday that all the
daqd issues were coming from the frame writing,
I started to dig into it more today. 
  
  11412   Tue Jul 14 16:51:01 2015 JamieSummaryCDSCDS upgrade: problem is not disk accessI think I have now determined once and
for all that the daqd problems are NOT due
to disk IO contention.
  
  11393   Tue Jul 7 18:27:54 2015 JamieSummaryCDSCDS upgrade: progress!After a couple of days of struggle, I made
some progress on the CDS upgrade today:

 2.9-RTS-OK.pngC1X04_GDS_TP.png 
  11415   Wed Jul 15 13:19:14 2015 JamieSummaryCDSCDS upgrade: reducing mx end-points as last ditch effortI tried one last thing, suggested by Keith
and Gerrit.  I tried reducing the number
of mx end-points on fb to zero, which should
  
  11417   Wed Jul 15 18:19:12 2015 JamieSummaryCDSCDS upgrade: tentative stabilty?Keith Thorne provided his eyes on the situation
today and had some suggestions that might
have helped things
  
  8547   Tue May 7 23:03:12 2013 KojiConfigurationCDSCDS workSummary:
c1rfm / c1lsc / c1ass / c1sus were
  
  11221   Wed Apr 15 20:54:18 2015 JenneUpdateComputer Scripts / ProgramsCDSutils upgrade badThe SUS align/misalign scripts don't
work after the new CDS utils upgrade. 

I don't know if it's looking
  
  11223   Wed Apr 15 23:29:08 2015 JenneUpdateComputer Scripts / ProgramsCDSutils upgrade undoneQ remotely reverted this change. 
Scripts seem to work again.

  
ELOG V3.1.3-