I’ll add a note here for future readers of the log, because previous logs on the subject of cymac1 OS upgrades may have made it appear a little too easy.

If you are reading this because you're having an issue with cymac1 (or another front-end computer), you may be wondering: Should I try upgrading the OS on cymac1 to fix my problem?

The answer to that question is: A resounding no!

An OS upgrade is almost never going to be the solution to any problem you may have with cymac1. In fact, it is almost guaranteed to create many new problems that you will have to solve before cymac1 becomes usable again.

Upgrading the OS on cymac1 should only be undertaken if you know exactly why it is absolutely necessary, AND when you have a lot of free time on your hands to troubleshoot the problems it will inevitably create, AND after you have verified there's a backup of the existing system that you can restore to if needed.

UPDATE: the system is running and acquiring data again (although a DAC status bit remains red). Things that had to be fixed to recover from this upgrade were the following:

1. update daqdrc config file, environment variables in env/systemd_env, and provide symbolic links to additional config files in /etc/advligorts
2. get new standalone_edc service running to pull in data from external epics channels
3. rtcds build and rtcds install all models (some had been built but not installed)
4. add advligorts user to controls group
5. add group write permissions to files under /opt/rtcds and /frames
6. get new local_dc service running, to combine data from all front ends for ingestion by daqd
7. disable mx_crc_only service that is no longer needed

Also to note, rtcds start was working to get front end codes started, but rtcds stop apparently fails to remove the front end modules sometimes (follow it with rmmod when that happens).