40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Thu Oct 6 12:02:21 2022, Anchal, Update, CDS, CDS Upgrade Plan 
    Reply  Fri Oct 7 22:45:15 2022, Anchal, Update, CDS, CDS Upgrade Status Update 
       Reply  Mon Oct 10 10:14:05 2022, Chris, Update, CDS, CDS Upgrade remaining issues timingslip.pngadctimingglitch.png
          Reply  Tue Oct 11 10:58:36 2022, Chris, Update, CDS, CDS Upgrade remaining issues PXL_20221010_212415180.jpg
             Reply  Tue Oct 11 11:27:58 2022, Koji, Update, CDS, CDS Upgrade remaining issues 
                Reply  Tue Oct 18 15:24:48 2022, Tega, Update, CDS, Rack 1X7 work proposal 
                   Reply  Thu Oct 20 19:47:24 2022, Tega, Update, CDS, Relocate front-ends to Rack 1X7 IMG_20221020_193623269.jpgIMG_20221020_193637034.jpgiop_status_20221020.pngiop_status_20221020_restart_models.pngc1ioo_shm_glitch.png
                      Reply  Wed Nov 9 14:19:18 2022, Tega, Update, CDS, fb1 OS migration 
                         Reply  Mon Nov 21 15:16:12 2022, Tega, Update, CDS, workstation upgrade 
                            Reply  Tue Nov 22 15:33:02 2022, Tega, Update, CDS, I/O chassis parts inventory 
                               Reply  Wed Nov 23 16:12:00 2022, Tega, Update, CDS, I/O chassis parts inventory II 
                            Reply  Mon Nov 28 16:58:20 2022, Paco, Update, CDS, pypi package added 
                               Reply  Wed Nov 30 14:25:34 2022, Tega, Update, CDS, AMD display driver installation 
Message ID: 17175     Entry time: Thu Oct 6 12:02:21 2022     Reply to this: 17178
Author: Anchal 
Type: Update 
Category: CDS 
Subject: CDS Upgrade Plan 

[Chris, Anchal]

Chris and I discussed our plan for CDS upgrade which amounts to moving new FEs, new chiara, and new FB1 OS system tomartian network.


Preparation:

  • Chiara (clone) (will be called "New Chiara" henceforth) will be resynced to existing chiara to get all model and medm changes.
  • All models on New Chiara will be rebuilt, and reinstalled.
  • All running servies on existing chiara will be printed and stored for comparison later.
  • New Chiara's OS drive will be updraged to Debian 10 and all services will be restored:
    • DHCP
    • DNS
    • NFS
    • rsync
  • Existing fb1 DAQ network card (10 GBps ethernet card) will be verified.
  • Make a list of all fb1 file system mounts and their UUIDs.

Upgrade plan:

Date: Fri Oct 7, 2022
Time: 11:00 am (After coffee and donuts)
Minimum required people: Chris, Anchal, JC (the more the merrier)

Steps:

  1. Ensure a snapshot of all channels is present from Oct 6th on New Chiara.
  2. Shutdown all machines:
    1. All slow computers (Except c1vac).
      Computer List: ssh into the computers and run:
      sudo systemctl stop modbusIOC.service
      sudo shutdown -h now
      1. c1susaux
      2. c1susaux2
      3. c1auxex
      4. c1auxey1
      5. c1psl
      6. c1iscaux
    2. All fast computers. Run on rossa:
      /cvs/cds/rtcds/caltech/c1/Git/40m/scripts/cds/stopAllModels.sh
      Disconnect left ethernet cables from the back of these computers.
    3. Power off all I/O chassis
    4. Swap the oneStop cables on all I/O chassis to fiber cables. On c1sus, connect the copper oneStop cable to teststand c1sus FE.
    5. Tun on all I/O chassis.
  3. Exchange chairas.
    1. Connect old chiara to teststand network.
    2. Connect New Chiara to martian network.
    3. Turn on both old and new chiara.
    4. Ensure all services are running on New Chiara by comparing with the list made earlier during preparation.
  4. fb1.
    1. Move fb1(clone)'s OS drive into existing fb1 (on 1X6)
    2. Turn on fb1 (on 1X6).
    3. Ensure fb1 is mounting all it's file systems correctly.
  5. New FEs
    1. Connect the network switch for new FEs to martian network. Make sure that old chiara is not connected to this same switch.
    2. Turn on the new FEs. All models should start on boot in sequence.
    3. Check if all models have green lights.
  6. Burt restore using latest snapshot available.
  7. Perform tests:
    1. Check if all local damping loops are working as before.
    2. Check if all IPC channels are transmitting and receiving correctly.
    3. Check if IMC is able to lock.
    4. Try single arm locking
    5. Try MICH locking.
  8. Make contingency plan on how to revert to old system if something fails.
ELOG V3.1.3-