40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Aug 4 19:01:59 2022, Tega, Update, Computers, Front-end machine in supermicro boxes IMG_20220804_184444473.jpgIMG_20220804_191658206.jpgIMG_20220804_185336240.jpgIMG_20220804_185023002.jpg
    Reply  Mon Aug 8 17:16:51 2022, Tega, Update, Computers, Front-end machine setup IMG_20220808_170349717.jpg
       Reply  Wed Aug 10 20:51:14 2022, Tega, Update, Computers, CDS upgrade Front-end machine setup IMG_20220810_171002928.jpgIMG_20220810_171019633.jpg
          Reply  Tue Aug 16 18:22:59 2022, Tega, Update, Computers, c1teststand rack mounting for CDS upgrade IMG_20220816_180157132.jpgIMG_20220816_175125874.jpg
             Reply  Wed Aug 17 11:10:51 2022, rana, Update, Computers, c1teststand rack mounting for CDS upgrade 
             Reply  Mon Aug 22 19:02:15 2022, Tega, Update, Computers, c1teststand rack mounting for CDS upgrade II IMG_20220822_185437763.jpgIMG_20220822_131340214.jpgc1bhd.jpegc1lsc.jpeg
                Reply  Tue Aug 23 22:30:24 2022, Tega, Update, Computers, c1teststand OS upgrade - I 
                Reply  Fri Aug 26 14:05:09 2022, Tega, Update, Computers, rack reshuffle proposal for CDS upgrade 6x
                   Reply  Sun Aug 28 23:14:22 2022, Jamie, Update, Computers, rack reshuffle proposal for CDS upgrade 
                      Reply  Mon Aug 29 15:15:46 2022, Tega, Update, Computers, 3 FEs from LLO got delivered today IMG_20220829_145533452.jpgIMG_20220829_144801365.jpg
                         Reply  Tue Aug 30 15:21:27 2022, Tega, Update, Computers, 3 FEs from LHO got delivered today IMG_20220830_144925325.jpgIMG_20220830_142307495.jpgIMG_20220830_143059443.jpg
                   Reply  Mon Sep 19 20:21:06 2022, Tega, Update, Computers, 1X7 and 1X6 work IMG_20220919_204013819.jpgIMG_20220919_203541114.jpgIMG_20220919_203458952.jpg
                      Reply  Tue Sep 20 23:06:23 2022, Tega, Update, Computers, Setup the 6 new front-ends to boot off the FB1 clone IMG_20220921_084220465.jpgdolphin_err_init_state.pngdolphin_final_state.png
                         Reply  Wed Sep 21 17:16:14 2022, Tega, Update, Computers, Setup the 6 new front-ends to boot off the FB1 clone 
                            Reply  Thu Sep 22 20:57:16 2022, Tega, Update, Computers, build, install and start 40m models on teststand  dolphin_state_plus_c1shimmer.pngFE_status_overview.png
                               Reply  Fri Sep 23 19:07:03 2022, Tega, Update, Computers, Work to improve stability of 40m models running on teststand  dolphin_state_all_green.pngdolphin_state_IPC_glitch.png
                            Reply  Thu Sep 29 15:12:02 2022, JC, Update, Computers, Setup the 6 new front-ends to boot off the FB1 clone 
                               Reply  Tue Oct 4 21:00:49 2022, Chris, Update, Computers, Failed takeover attempt with the new front ends 
                                  Reply  Thu Oct 6 07:29:30 2022, Chris, Update, Computers, Successful takeover attempt with the new front ends 
Message ID: 17172     Entry time: Tue Oct 4 21:00:49 2022     In reply to: 17164     Reply to this: 17173
Author: Chris 
Type: Update 
Category: Computers 
Subject: Failed takeover attempt with the new front ends 

[Jamie, JC, Chris]

Today we made a failed attempt to take over the 40m hardware with the new front ends on the test stand.

As an initial test, we connected the new c1iscey to its I/O chassis using the OneStop fiber link. This went smoothly, so we tried to proceed with the rest of the system, which uncovered several problems. Consequently, we’ve reverted control back to the old front ends tonight, and will regroup and make another attempt tomorrow.

Status summary:

  • c1iscey worked on the first try
  • c1lsc worked, after we sorted out which of the two OneStop cables run to its rack we needed to use
  • c1sus2 sort of worked (its models have been crashing sporadically)
  • c1ioo had a busted OneStop cable, and worked after that was replaced
  • c1sus refused to work with the fiber OneStop cables (we tried several, including the known working one from c1ioo), but we jury-rigged it to run over a copper cable, after nudging the teststand rack a bit closer to the chassis
  • c1iscex refused to work with the fiber OneStop cables, and substituting copper was not an option, so we were stuck

There are various pathologies that we've seen with the OneStop interface cards in the I/O chassis. We don't seem to have the documentation for these cards, but our interpretive guesses are as follows:

  • When working, it is supposed to illuminate all the green LEDs along the top of the card, and the four next to the connector. In this state, you can run lspci -vt on the host, and see the various PLX/Contec/etc devices that populate the chassis.
  • When the cable is unplugged or bad, only four green LEDs illuminate on the card, and none by the connector. No devices from the chassis can be seen from the host.
  • On c1iscex and c1sus, when a fiber link is plugged in, it turns on all the LEDs along the top of the card, but the four next to the connector remain dark. We’re not sure yet what this is trying to tell us, but lspci finds no devices from the chassis, same as if it is unplugged.
  • Also, sometimes on c1iscex, no LEDs would illuminate at all (possibly the card was not seated properly).

Tomorrow, we plan to swap out the c1iscex I/O chassis for the one in the test stand, and see if that lets us get the full system up and running.

ELOG V3.1.3-