40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  Cryo Lab eLog  Not logged in ELOG logo
Entry  Tue Feb 2 09:52:19 2021, aaron, Computing, DAQ, cymac to Debian buster 
    Reply  Wed Feb 3 08:23:12 2021, Chris, Computing, DAQ, cymac to Debian buster 
       Reply  Tue Feb 9 16:33:31 2021, aaron, Computing, DAQ, cymac to Debian buster 
          Reply  Thu Feb 11 18:51:39 2021, aaron, Computing, DAQ, cymac to Debian buster Screen_Shot_2021-02-11_at_19.27.21.png
             Reply  Mon Feb 15 18:25:19 2021, Chris, Computing, DAQ, cymac to Debian buster 
                Reply  Tue Feb 16 14:29:04 2021, aaron, Computing, DAQ, cymac to Debian buster 55B5D301-4243-45DA-9B03-45EF87B2A684.jpeg4C7FE7DC-E8A0-4770-9A49-14EB491F76E2.jpeg13D67AB6-5A77-4C94-A2A4-FE7FDF065B01.jpegC84B3054-269B-4969-817F-1429BB671C7A.jpeg
                   Reply  Thu Feb 18 16:16:38 2021, aaron, Computing, DAQ, cymac to Debian buster 15A01777-A1C5-4710-8180-28C8B68A07AE.jpegF90CD5EF-1A26-4C4A-9C84-046E49642545.jpeg
                      Reply  Fri Feb 19 14:43:09 2021, aaron, Computing, DAQ, AI chassis diagnosis 
                   Reply  Tue Mar 9 14:03:12 2021, aaron, Computing, DAQ, oma model 
Message ID: 2624     Entry time: Tue Feb 2 09:52:19 2021     Reply to this: 2625
Author: aaron 
Type: Computing 
Category: DAQ 
Subject: cymac to Debian buster 

enter Tue Feb 2 09:52:17 2021

setting up x1oma model

Last time, I ran into symlinking or kernel issues.

on cymac:

sudo apt-get update

Get an error message: GPG error about out of date signatures in the repo, and/or the necessary public key is not available. The error message points to the apt repo's readme, with instructions to add a line to /etc/apt/sources.list.d/lscsoft.list. I uncomment from that file:

deb http://software.ligo.org/lscsoft/debian stretch contrib

but get the same error messages. some of the errors are for stretch-legacy, others for stretch.

I haven't used the realtime system since last year Chris upgraded cymac1's Debian distro from 8 (jessie) to 9 (stretch), and the slow machines to 10 (buster). So, I figure I'll go through the rtcds wiki for setting up cymac with a fine comb, starting by enabling CDSSoftware repos. Buster is now the fully supported Debian release, so to keep all our machines on the same page I'm upgrading cymac1 to Buster. 

install buster

  • Backed up the recommended repos to spirou
  • remove extra cds/lsc software in /etc/apt/sources.list.d (also backed up to spirou), and replace references to stretch in /etc/apt/sources.list to buster
  • apt update to latest version of stretch buster
  • apt-get upgrade to take care of packages that don't require new packages to be removed or installed, then apt full-upgrade to take care of the rest
  • apt install linux-image-4.19-rt-amd64 (hm... was this necessary or desirable? I ended up installing linux-image-amd64 instead, and only later realized that the cymac setup instructions have a unique kernel--see below. Maybe this is related to the kernel errors I was getting earlier?)
  • reboot

Returning to installing cds software:

make a new /etc/apt/sources.list.d/lscsoft.list and cdssoft.list, same as the old but replacing 'stretch' with 'buster'.

apt-get update
apt-get -y --force-yes install lscsoft-archive-keyring
wget -c http://apt.ligo-wa.caltech.edu/debian/pool/buster-unstable/cdssoft-release-buster/cdssoft-release-buster_1.0.5_all.deb
sudo dpkg -i cdssoft-release-buster_1.0.5_all.deb
sudo apt update
sudo apt upgrade  # ifo set to TST and site set to x1 # this step not necessary
sudo apt install cds-workstation # ERROR: dependencies on cds-crtools, ndscope, and python3-nds2-client can't be resolved. error resolved with correction to buster install steps bolded above
# in /etc/apt/sources.list.d/cdssoft-restricted.list add the line
# deb [signed-by=/usr/share/keyrings/cdssoft-unstable-archive-keyring.gpg] https://$USER:$PASSWORD@apt.ligo-wa.caltech.edu/debian buster-restricted main

setting up cymac

sudo apt install advligorts-cymac # same model directory and user
sudo install linux-image-4.19.0-6-rtcds-amd64-unsigned # ERROR: missing destination file operand, the wiki seems to omit apt
sudo apt install linux-image-4.19.0.6-rtcds-amd64 unsigned

The menu structure for this new rtcds kernel is "Debian GNU/Linux>Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 4.19.0-6-rtcds-amd64", which I copy into /etc/default/grub, followed by

sudo update-grub
reboot

Odd, I'm still seeing the old 4.19.0-14-amd64 when running uname -r... which I uninstalled. Removing it again, repeating update-grub, and reboot did the trick. Installation complete.

The only error I'm still seeing is "Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://apt.ligo-wa.caltech.edu/debian buster-restricted InRelease' doesn't support architecture 'i386'". This is odd, because I'm running Debian 10 on an amd64 machine, with an amd64 kernel; I last saw this error when configuring cominaux. I think it's because I 'remove'd instead of 'purge'd the old kernels. After purging the non-rtcds kernels, I reboot and... no change in error message. This is not a problem, there is an i386 architecture on cymac1 for running zurich instrument tools, but this shouldn't affect the amd64 architecture; error only occurs for cdssoft-restricted.

outcome

model frontends aren't running. Lab snacks are missing. Outlook grim.

  • Run /opt/rtcds/txt/x1/scripts/startAll.sh. Script runs without error, but I still can't connect to cymac1 through nds.
  • rtcsd status tells me all model are not loaded, and the overall status is 'degraded'. Some kernel modules are missing.
  • rtcds start --all... no change
  • Jon set up a cymac in QIL recently
  • After running systemctl reset-failed, systemctl reports overall state 'running', but running s/startAll.sh gives warnings about identical process variable names on multiple servers. I think reset-failed just silences warnings with no change in state. 
  • /etc/advligorts/env has the same environment variables as before, so problem isn't there
  • This entry from Chris suggests to me perhaps rebuilding all models will help, since they were originally compiled with the old rts
    • rtcds make --all
      rtcds install --all
      reboot
    • no change in systemctl status

  • Investigating the failed systemctl units, I see that rts-daqd is among the units that failed to activate. That's a critical unit, so I inquire with journalctl _PID=1140 (the PID # of the rts-daqd process is listed with systemctl status rts-daqd). The failure is at
    • set master_config="/etc/advligorts/master" 
    • This makes sense, but is surprising--there is no such file, on cymac1 or in my backup! I hadn't heard of this file in any of the rts documentation. The only mention I can find is in a git issue.

It's nearly time for me to hand the lab over to Shruti, so I will stop for now.

exit Tue Feb 2 17:55:24 2021

ELOG V3.1.3-