40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Message ID: 15276     Entry time: Fri Mar 13 20:00:50 2020
Author: Jon 
Type: Update 
Category: Computers 
Subject: Loopback monitoring for slow machines 

Summary

Today I finished implementing loopback monitors of the up/down state of the slow controls machines. They are visible on a new MEDM screen accessible from Sitemap > CDS > Slow Machine Status (pictured in attachment 1). Each monitor is a single EPICS binary channel hosted by the slow machine, which toggles its state at 1 Hz (an alive "blinker"). For each machine, the monitor is defined by a separate database file named c1[machine]_state.db located in the target directory.

This is implemented for all upgraded machines, which includes every slow machine except for c1auxey. This is the next and final one slated for replacement.

Implementation

The blinkers are currently implemented as soft channels, but I'd like to ultimately convert them to hard channels using two sinking/sourcing BIO units. This will require new wiring inside each Acromag chassis, however. For now, as soft channels, the monitors are sensitive to a failure of the host machine or a failure of the EPICS IOC. As hard channels, they will additionally be sensitive to a failure of the secondary network interface, as has been known to happen.

Each slow machine's IOC had to be restarted this afternoon to pick up the new channels. The IOCs were restarted according to the following procedure.

c1auxex

  • Disabled OPLEV servos on ETMX
  • Zeroed slow biases
  • Disabled watchdog
  • Restarted IOC
  • Reverted 1-3

​c1vac

  • Closed V1, VM1
  • Restarted IOC
  • Returned valves to original state

c1psl

  • Disabled IMC autolocker
  • Closed PSL shutter
  • Restarted IOC
  • Reverted 1-2

c1iscaux

  • ​Restarted IOC

c1susaux

  • Disabled IMC autolocker
  • Closed shutter
  • Disabled OPLEV servos on: MC1, MC2, MC3, BS, ITMX, ITMX, PRM, SRM
  • Zeroed slow biases
  • Disabled watchdogs
  • Restarted IOC
  • Reverted 1-5

The intial recovery of c1susaux did not succeed. Most visibly, the alignment state of the IFO was not restored. After some debugging, we found that the restart of the modbus service was partially failing at the final burt-restore stage. The latest snapshot file /opt/rtcds/caltech/c1/burt/autoburt/latest/c1susaux.snap was not found. I manually restored a known good snapshot from earlier in the day (15:19) and we were able to relock the IMC and XARM. GV and I were just talking earlier today about eliminating these burt-restores from the systemd files. I think we should.

Attachment 1: Screen_Shot_2020-03-13_at_7.59.55_PM.png  14 kB  Uploaded Fri Mar 13 21:02:51 2020  | Hide | Hide all
Screen_Shot_2020-03-13_at_7.59.55_PM.png
ELOG V3.1.3-