The local backup seems working fine again. But I found that megatron is down and this is a real issue. This should be fixed at the earliest chance.
It seems that the local backup has been successfully taken this morning.
controls@nodus|backup> tail /opt/rtcds/caltech/c1/scripts/backup/localbackup.log
2021-12-19 07:00:01,146 INFO Updating backup image of /cvs/cds
2021-12-19 07:00:01,146 ERROR External drive not mounted!!!
2021-12-20 07:00:01,255 INFO Updating backup image of /cvs/cds
2021-12-20 07:00:01,255 ERROR External drive not mounted!!!
2021-12-21 07:00:01,361 INFO Updating backup image of /cvs/cds
2021-12-21 07:00:01,361 ERROR External drive not mounted!!!
2021-12-22 07:00:01,469 INFO Updating backup image of /cvs/cds
2021-12-22 07:00:01,470 ERROR External drive not mounted!!!
2021-12-23 07:00:01,594 INFO Updating backup image of /cvs/cds
2021-12-23 07:19:55,560 INFO Backup rsync job ran successfully, transferred 338425 files.
However, I noticed that the autoburt has been stalled since Dec 6 (I used to check how the backup is up-to-date using the autoburt snapshots)
Dec>pwd
/opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/Dec
Dec>ls -l
total 24
drwxr-xr-x 26 controls controls 4096 Dec 1 23:07 1
drwxr-xr-x 26 controls controls 4096 Dec 2 23:07 2
drwxr-xr-x 26 controls controls 4096 Dec 3 23:07 3
drwxr-xr-x 26 controls controls 4096 Dec 4 23:07 4
drwxr-xr-x 26 controls controls 4096 Dec 5 23:07 5
drwxr-xr-x 19 controls controls 4096 Dec 6 16:07 6
There are a bunch of errors in the log file as follows, but maybe this is not an issue
controls@nodus|burt> pwd
/opt/rtcds/caltech/c1/burt
controls@nodus|burt> tail burtcron.log
!!! ERROR !!! Target c1supepics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1tstepics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1x10epics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1aux Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1dcuepics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1iscaux Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1iscepics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1losepics Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1psl Snapshot file inconsistent with Request file
!!! ERROR !!! Target c1susaux Snapshot file inconsistent with Request file
The real issue seems that megatron is down. It has a lot of house keeping jobs on corn including the N2 pressure alert.
https://wiki-40m.ligo.caltech.edu/Computers_and_Scripts/CRON
This needs to be fixed at the earliest chance. |