I read on some stack exchange that 'NTP synchornized' indicator turns 'yes' in the output of command timedatectl only when RTC clock has been adjusted at some point. I also read that timesyncd does not do the change if the time difference is too much, roughly more than 3 seconds.

So I logged into all FE machines and ran sudo hwclock -w to synchronize them all to the system clocks and then waited if the timesyncd does any correction on RTC. It did not. A few hours later, I found the RTC clocks drifitng again from the system clocks. So even if the timesynd service is running as it should, it si not performing time correction for whatever reason.

Maybe we should try to use some other service?

Related elog thread: 16286

I didn't really achieve anything but I'm listing what I've tried.

• I know now that the timesyncd isn't working because systemd-timesyncd is known to have issues when running on a read-only file system. In particular, the service does not have privileges to change the clock or drift settings at /run/systemd/clock or /etc/adjtime.
• The workarounds to these problems are poorly rated/reviews in stack exchange and require me to change the /etc/systmd/timesyncd.conf file but I'm unable to edit this file.
• I know that Paco was able to change these files earlier as the files are now changed and configured to follow a debian ntp pool server which won't work as the FEs do not have internet access. So the conf file needs to be restored to using ntpserver as the ntp server.
• From system messages, the ntpserver is recognized by the service as shown in the second part of 16285. I really think the issue is in file permissions. the file /etc/adjtime has never been updated since 2017.
• I got help from Paco on how to edit files for FE machines. The FE machines directories are exported from fb1:/diskless/root.jessie/
• I restored the /etc/systmd/timesyncd.conf file to how it as before with just servers=ntpserver line. Restarted timesyncd service on all FEs,I tried a few su the synchronization did not happen.
• I tried a few suggestions from stackexchange but none of them worked. The only rated solution creates a tmpfs directory outside of read-only filesystem and uses that to run timesyncd. So, in my opinion, timesyncd  would never work in our diskless read-only file system FE machines.
• One issue in an archlinux discussion ended by the questioner resorting to use opennptd from openBSD distribution. The user claimed that opennptd is simple enough that it can run ntp synchornization on a read-only file system.
• Somehwat painfully, I 'kind of' installed the openntpd tool in the fb1:/diskless/root.jessie directory following directions from here. I had to manually add user group and group for the FEs (which I might not have done correctly). I was not able to get the openntpd daemon to start properly after soe tries.
• I restored everything back to how it was and restarted timesyncd in c1sus even though it would not do anything really.

tl;dr: NTP servers and clients were never synchronized, are not synchronizing even with ntp... nodus is synchronized but uses chronyd; should we use chronyd everywhere?

Spent some time investigating the ntp synchronization. In the morning, after Anchal set up all the ntp servers / FE clients I tried restarting the rts IOPs with no success. Later, with Tega we tried the usual manual matching of the date between c1iscex and fb1 machines but we iterated over different n-second offsets from -10 to +10, also without success.

This afternoon, I tried debugging the FE and fb1 timing differences. For this I inspected the ntp configuration file under /etc/ntp.conf in both the fb1 and /diskless/root.jessie/etc/ntp.conf (for the FE machines) and tried different combinations with and without nodus, with and without restrict lines, all while looking at the output of sudo journalctl -f on c1iscey. Everytime I changed the ntp config file, I restarted the service using sudo systemctl restart ntp.service . Looking through some online forums, people suggested basic pinging to see if the ntp servers were up (and broadcasting their times over the local network) but this failed to run (read-only filesystem) so I went into fb1, and ran sudo chroot /diskless/root.jessie/ /bin/bash to allow me to change file permissions. The test was first done with /bin/ping which couldn't even open a socket (root access needed) by running chmod 4755 /bin/ping then ssh-ing into c1iscey and pinging the fb1 machine successfully. After this, I ran chmod 4755 /usr/sbin/ntpd so that the ntp daemon would have no problem in reaching the server in case this was blocking the synchronization. I exited the chroot shell and the ntp daemon in c1iscey; but the ntpstat still showed unsynchronised status. I also learned that when running an ntp query with ntpq -p if a client has succeeded in synchronizing its time to the server time, an asterisk should be appended at the end. This was not the case in any FE machine... and looking at fb1, this was also not true. Although the fb1 peers are correctly listed as nodus, the caltech ntp server, and a broadcast (.BCST.) server from local time (meant to serve the FE machines), none appears to have synchronized... Going one level further, in nodus I checked the time synchronization servers by running chronyc sources the output shows

controls@nodus|~> chronyc sources
210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* testntp1.superonline.net      1  10   377   280  +1511us[+1403us] +/-   92ms
^+ 38.229.59.9                   2  10   377   206  +8219us[+8219us] +/-  117ms
^+ tms04.deltatelesystems.ru     2  10   377   23m    -17ms[  -17ms] +/-  183ms
^+ ntp.gnc.am                    3  10   377   914  -8294us[-8401us] +/-  168ms

I then ran chronyc clients to find if fb1 was listed (as I would have expected) but the output shows this --

Hostname                   Client    Peer CmdAuth CmdNorm  CmdBad  LstN  LstC
=========================  ======  ======  ======  ======  ======  ====  ====
501 Not authorised

So clearly chronyd succeeded in synchronizing nodus' time to whatever server it was pointed at but downstream from there, neither the fb1 or any FE machines seem to be synchronizing properly. It may be as simple as figuring out the correct ntp configuration file, or switching to chronyd for all machines (for the sake of homogeneity?)

I attempted to install chrony and run it on one of the FE machines. It didn't work and in doing so, I lost the working NTP client service on the FE computers as well. Following are some details:

• I added the following two mirrors in the apt source list of root.jessie at /etc/apt/sources.list
  deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
  deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free
• Then I installed chrony in the root.jessie using
  sudo apt-get install chrony
  • I was getting an error E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory) . To fix this, I had to run:
    sudo mount -t devpts none "$rootpath/dev/pts" -o ptmxmode=0666,newinstance
    sudo ln -fs "pts/ptmx" "$rootpath/dev/ptmx"
  • Then, I had another error to resolve.
    Failed to read /proc/cmdline. Ignoring: No such file or directory
    start-stop-daemon: nothing in /proc - not mounted?
    To fix this, I had to exit to fb1 and run:
    sudo mount --bind /proc /diskless/root.jessie/proc
  • With these steps, chrony was finally installed, but I immediately saw an error message saying:
    Starting /usr/sbin/chronyd...
    Could not open NTP sockets
• I figured this must be due to ntp running in the FE machines.  I logged into c1iscex and stopped and disabled the ntp service:
  sudo systemctl stop ntp
  sudo systemctl disable ntp
  • I saw some error messages from the above coomand as FEs are read only file systems:
    Synchronizing state for ntp.service with sysvinit using update-rc.d...
    Executing /usr/sbin/update-rc.d ntp defaults
    insserv: fopen(.depend.stop): Read-only file system
    Executing /usr/sbin/update-rc.d ntp disable
    update-rc.d: error: Read-only file system
  • So I went back to chroot in fb1 and ran the two command sabove that failed:
    /usr/sbin/update-rc.d ntp defaults
    /usr/sbin/update-rc.d ntp disable
  • The last line gave the output:
    insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
    insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
  • I igored this and moved forward.
• I copied the chronyd.service from nodus to the chroot in fb1 and configured it to use nodus as the server. The I started the chronyd.service

  sudo systemctl status chronyd.service
  but got the saem issue of NTP sockets.

  â—Â� chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
     Active: failed (Result: exit-code) since Tue 2021-08-24 21:52:30 PDT; 5s ago
    Process: 790 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=1/FAILURE)

  Aug 24 21:52:29 c1iscex systemd[1]: Starting NTP client/server...
  Aug 24 21:52:30 c1iscex chronyd[790]: Could not open NTP sockets
  Aug 24 21:52:30 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
  Aug 24 21:52:30 c1iscex systemd[1]: Failed to start NTP client/server.
  Aug 24 21:52:30 c1iscex systemd[1]: Unit chronyd.service entered failed state.

• I tried a few things to resolve this, but couldn't get it to work. So I gave up on using chrony and decided to go back to ntp service atleast.

• I stopped, disabled and checked status of chrony:
  sudo systemctl stop chronyd
  sudo systemctl disable chronyd
  sudo systemctl status chronyd
  This gave the output:

  â—Â� chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
     Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:07 PDT; 25s ago

  Aug 24 22:09:07 c1iscex systemd[1]: Starting NTP client/server...
  Aug 24 22:09:07 c1iscex chronyd[2490]: Could not open NTP sockets
  Aug 24 22:09:07 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
  Aug 24 22:09:07 c1iscex systemd[1]: Failed to start NTP client/server.
  Aug 24 22:09:07 c1iscex systemd[1]: Unit chronyd.service entered failed state.
  Aug 24 22:09:15 c1iscex systemd[1]: Stopped NTP client/server.

• I went back to fb1 chroot and removed chrony package and deleted the configuration files and systemd service files:
  sudo apt-get remove chrony

• But when I started ntp daemon service back in c1iscex, it gave error:
  sudo systemctl restart ntp
  Job for ntp.service failed. See 'systemctl status ntp.service' and 'journalctl -xn' for details.

• Status shows:

  sudo systemctl status ntp
  â—Â� ntp.service - LSB: Start NTP daemon
     Loaded: loaded (/etc/init.d/ntp)
     Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:56 PDT; 9s ago
    Process: 2597 ExecStart=/etc/init.d/ntp start (code=exited, status=5)

  Aug 24 22:09:55 c1iscex systemd[1]: Starting LSB: Start NTP daemon...
  Aug 24 22:09:56 c1iscex systemd[1]: ntp.service: control process exited, code=exited status=5
  Aug 24 22:09:56 c1iscex systemd[1]: Failed to start LSB: Start NTP daemon.
  Aug 24 22:09:56 c1iscex systemd[1]: Unit ntp.service entered failed state.

• I tried to enable back the ntp service by sudo systemctl enable ntp. I got similar error messages of read only filesystem as earlier.
  Synchronizing state for ntp.service with sysvinit using update-rc.d...
  Executing /usr/sbin/update-rc.d ntp defaults
  insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
  insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
  insserv: fopen(.depend.stop): Read-only file system
  Executing /usr/sbin/update-rc.d ntp enable
  update-rc.d: error: Read-only file system

  • I went back to chroot in fb1 and ran:
    /usr/sbin/update-rc.d ntp defaults
    insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
    insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
    and
    /usr/sbin/update-rc.d ntp enable

• I came back to c1iscex and tried restarting the ntp service but got same error messages as above with exit code 5.

• I checked c1sus, the ntp was running there. I tested the configuration by restarting the ntp service, and then it failed with same error message. So the remaining three FEs, c1lsc, c1ioo and c1iscey have running ntp service, but they won't be able to restart.

• As a last try, I rebooted c1iscex to see if ntp comes back online nicely, but it doesn't.

Bottom line, I went to try chrony in the FEs, and I ended up breaking the ntp client services on the computers as well. We have no NTP synchronization in any of the FEs.

Even though Paco and I are learning about the ntp and cds stuff, I think it's time we get help from someone with real experience. The lab is not in a good state for far too long.

Jamie told me to use chroot to log in into the chroot jail of debian os that are exported for the FEs and install ntp there. I took following steps at the end of which, all FEs have NTP synchronized now.

• I logged into fb1 through nodus.
• chroot /diskless/root.jessie /bin/bash took me to the bash terminal for debian os that is exported to all FEs.
• Here, I ran sudo apt-get install ntp which ran without any errors.
• I then edited the file in /etc/ntp.conf , i removed the default servers and added following lines for servers (fb1 and nodus ip addresses):
  server 192.113.168.201
  server 192.113.168.201
• I logged into each FE machine and ran following commands:
  sudo systemctl stop systemd-timesyncd.service; sudo systemctl status systemd-timesyncd.service;
  timedatectl; sleep 2;sudo systemctl daemon-reload;  sudo systemctl start ntp; sleep 2; sudo systemctl status ntp; timedatectl
  sudo hwclock -s
  • The first line ensures that systemd-timesyncd.service is not running anymore. I did not uninstall timesyncd and left its configuration file as it is.
  • The second line first shows the times of local and RTC clocks. Then reloads the daemon services to get ntp registered. Then starts ntp.service and shows it's status. Finally, the timedatectl command shows the synchronized clocks and that NTP synchronization has occured.
  • The last line sets the local clock same as RTC clock. Even though this wasn't required as I saw that the clocks were already same to seconds, I just wanted a point where all the local clocks are synchronized to the ntp server.
• Hopefully, this would resolve our issue of restarting the models anytime some glitch happens or when we need ot update something in one of them.

Edit Tue Aug 24 10:19:11 2021:

I also disabled timesyncd on all FEs using sudo systemctl disable systemd-timesyncd.service

I've added this wiki page for summarizing the NTP synchronization knowledge.

Crontab: The bug of data only plotting until 5PM is being investigated.  The crontab's final call to the summary page generator was at 5PM.  This means that the data plots were not being generated after 5PM, so clearly they never contained data from after 5PM.  In fact, the original crontab reads:

0 11,5,11,17 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1

I'm not exactly sure what inspired these entries.  The 11,5,11,17 entries are supposed to be the hours at which the program is run.  Why is it run twice at 11?  I assume it was just a typo or something.

The final call time was changed to 11:59PM in an attempt to plot the entire day's data, but this method didn't appear to work because the program would still be running past midnight, which was apparently inhibiting its functionality (most likely, the day change was affecting how the data is fetched).  The best solution is probably to just wait until the next day, then call the summary page generator on the previous day's data.  This will be implemented soon.

Calendars: Although the calendar tabs on the left side of the page were fixed, the calendars displayed at: https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/ appear to still have squished together text.  The calendar is being fetched from https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/calendar.html and displayed in the page.  This error is peculiar because the URL from which the calendar is being fetched does NOT have squished together text, but the resulting calendar at 40m-summary/calendar/ will not display spaces between the text.  This issue is still being investigated.

Main Problem:

The frequency counter (FC) takes in an analog RF input(signal) and outputs the frequency of the signal(Ranging from 1 MHz- 6000 MHz) in the digital domain (into a processor). The FC samples the data with a given sample rate( user defined) which ranges from 0.1 s to 1 s(faced problems in fixing this initially).  For data acquisition, we have been using a Raspberry Pi(as a processor) which is connected to the martian network and can communicate with the computers inside the 40m.  The ultimate challenge which I faced( and been knocking my head off from past two-three weeks) is the synchronization of clocks between the Raspberry Pi and the FC i.e the clock which the FC uses to sample and dump data( every 'x' s) and the clock inside the raspberry pi( used  in the loop to wait for a particular amount of time the frequency counter takes to dump successive data).

Steps Taken:

• To address this problem, first I added an external clock circuit which monitors the Raspberry Pi and the FC to dump and read data at a particular rate(which is equal to the sampling rate of the FC)In detail: http://nodus.ligo.caltech.edu:8080/40m/10129. 
• While doing so, at first the level trigger algorithm was used which means that the external clock frequency was half as that of  the reciprocal of the sampling rate and a trigger was seen every time the level shifts from +DC to -DC(of the external square wave).
• But this did not completely mitigate the issues and there were still few issues on how quickly the ADC reads the signal and R Pi processes it.
• To minimize these issues completely, an edge trigger algorithm which detects a pos edge(rising)  of the clock was used. The clock  frequency is now equal to the reciprocal of the sampling rate. This algorithm showed better results and greatly minimized the drift of the sampling time.

Psuedo Code(code attached):

open device : FC via USB-HID;

open device : ADC via I2C;

always(for t= recording time):

            read data from ADC(external clock);

            if pos edge detected:

                    read data from FC and store it in a register;

             else read data from ADC;

end

write data stored in the register to a file( can be an Epics channel or a text file);

Results:

The attached are the plots showing the time between samples for a large number of samples taken for different sampling times of the FC. The percentage error is the percentage of standard error in the timing between two samples for the data for the entire measurement. It can be inferred that this error has been cut down to the order of ms.

To do next:

• I have started taking phase measurements( analysis and plots will follow this elog) and also the PSD plots with the improved timing characteristics.

Looks good. Now you have the internal timer to verify the external clock.
If you can realize the constant rate sampling without employing the external clock, that's quite handy.

The attached are the PSD plots with improved FC timing(with the same code as in http://nodus.ligo.caltech.edu:8080/40m/10151). More plots(Phase and PSD) to follow.

At the request of people down at LLO I've been trying to work on the reliability and speed of the GigE camera code.  In my testing, after several hours, the code would tend to lock up on the camera end.  It was also reported at LLO after several minutes the camera display would slow down, but I haven't been able to replicate that problem.

I've recently added some additional error checking and have updated to a more recent SDK which seems to help.  Attached are two plots of the frames per second of the code.  In this case, the frames per second  are measured as the time between calls to the C camera code for a new frame for gstreamer to encode and transmit.  The data points in the first graph are actually the averaged time for sets of 1000 frames.  The camera was sending 640x480 pixel frames, with an exposure time of 0.01 seconds.  Since the FPS was mostly between 45 and 55, it is taking the code roughly 0.01 second to process, encode, and transmit a frame.

During the test, the memory usage by the server code was roughly 1% (or 40 megabytes out of 4 gigabytes) and 50% of a CPU (out a total of  CPUs).

See Attachment #1. J8 was connected to a "LASTI timing slave" sitting in the rack that Chiara lives in - we don't use this for anything and I confirmed that there was no effect on the RTCDS when I pulled that fiber out. The LASTI timing slave also had a blinky that was blinking when the fiber was plugged in - which I take to believe that the slot works. 

Can we get away with just using these two available slots, J8 and J13? Do we really need three new expansion chassis?

I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.

That's great. I wonder if we can also get away with not adding new Dolphin infrastructure. I'd really like to avoid changing any IPC drivers.

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

That's great news we won't have to worry about a new timing fanout for the two new machines, c1bhd and c1sus2. And there's no plan to change Dolphin IPC drivers. The plan is only to install the same (older) version of the driver on the two new machines and plug into free slots in the existing switch.

Problem:

We had timing problems across the front ends.

Solution:

Noticing that the 1PPS reference was not blinking on the Master Timing Distribution box.  It was supposed to be getting a signal from the c0dcu1 VME crate computer, but this was not happening.

We disconnected the timing signal going into c0dcu1, coming from c0daqctrl, and connected the 1PPS directly from c0daqctrl to the Ref In for the Master Timing distribution box (blue box with lots of fibers coming out of it in 1X5).

We now have agreement in timing between front ends.

After several reboots we now have working RFM again, along with computers who agree on the current GPS time along with the frame builder.

Status:

RFM is back and testpoints should be happy.

We still don't have a working binary output for the X end.  I may need to get a replacement backplane with more than 4 slots if the 1st slot of this board has the same problem as the large boards.

I have burt restored the c1ioo, c1mcs, c1rms, c1sus, and c1scx processes, and optics look to be damped.

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

• Now, old frame data is no longer being overwritten
• For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

• The usual FE/FB power cycling did not fix the problem.
• The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
• This was off by 2 years.
• The way this worked up till now was by adding a fixed offset to this time.
  • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
  • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
  • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

1. First, we tried changing the offset in the daqdrc.fw file only.
   • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
   • This did not fix the problem.
2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
   • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
   • Added the following lines:
     /* 2018 had no leap seconds or leap days, so adjust for that */
            pHardware->gpsOffset += 31536000;
   • re-built and installed the modified symmetricom driver.
   • Checked that cat /proc/gps now yields the correct time.
   • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
   • With these steps, the frames were being written to /frames with the correct timestamp.
3. Next, we checked the timing on the FEs
   • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
   • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
   • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
   • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
   • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Summary:

The gps time mismatch between /proc/gps and gpstime seems to be resolved. However, the 0x4000 DC errors still persist. It is not clear to me why.

Details:

On the phone with J Hanks on Friday, he reminded me that c1sus seems to be the only machine with an IRIG-B timing card installed. I can't find the elog but I remembered that Jamie, ericq and I had done this work in 2016 (?), and I also remembered Jamie saying it wasn't working exactly as expected. Since the DAQ was working fine before this card was installed, and since there are no problems with the recording of channels from the other four FE machines without this card installed, I decided to simply pull out the card from the expansion chassis. The card has been stored in the CDS/FE cabinet along the Y arm for now. There was also a cable that interfaces to the card which brings over the 1pps from the GPS unit, which has also been stored in the CDS/FE cabinet.

This seems to have resolved the mismatch between the gpstime reported by cat /proc/gps and the gpstime commands - Attachment #1 (the <1 second mismatch is presumably due to the deadtime between commands). However, the 0x4000 DC errors still persist. I'll try the full power cycle of FEs and FB which has fixed this kind of error in the past, but apart from that, I'm out of ideas.

Update 1215:

Following the instructions in this elog did not fix the problem. The problem seems to be with the daqd_fw service, which reports the following:

Update 1530:

The frame-writer error was tracked down to a C0EDCU issue. Jon told me that the Hornet CC1 pressure gauge channel was renamed to . C1:Vac-CC1_pressure, and I made the change in the C0EDCU file. However, it returns a value of 9990000000.0, which the frame writer is not happy about... Keeping the old channel name makes the frame-writer run again (although the actual data is bunk).

Update 1755:

J Hanks suggested adding a 1 second offset to the daqdrc config files. This has now fixed the 0x4000 errors, and we are back to the "nominal" RTCDS status screen now - Attachment #2.

I started a GPS timing monitor script, /opt/rtcds/caltech/c1/Git/40m/scripts/cds/tempusTimingMon/tempusTimingMon.py, which runs inside a docker container on optimus. It accesses the GPS receiver (EndRun Tempus LX) status via pysnmp, and creates the following epics channels with pcaspy:

• C1:DAQ-GPS_TFOM “The Time Figure of Merit (TFOM) value ranges from 3 to 9 and indicates the current estimate of the worst case time error. It is a logarithmic scale, with each increment indicating a tenfold increase in the worst case time error boundaries.” (nominally 3, corresponding to 100 nsec)
• C1:DAQ-GPS_STATE “Current GPS signal processor state. One of 0, 1 or 2, with 0 being the acquisition state and 2 the fully locked on state.”
• C1:DAQ-GPS_SATS “Current number of GPS satellites being tracked.”
• C1:DAQ-GPS_DAC “Current 16 bit, Voltage Controlled TCXO DAC value. Typical range is 20000 to 40000, where more positive numbers have the effect of raising the TCXO frequency.”
• C1:DAQ-GPS_SNR “The current average carrier to noise ratio of all tracked satellites, in units of dB. Values less than 35 indicate weak signal conditions.”

Attached is a 1-day trend of the above channels, along with the IRIG-B timing offsets from the IOPs. No big timing excursions or slippages were seen yet, although the c1sus IOP (FEC-20) timing seems to be hopping around by one IOP sample time (15 µsec).

While recovering from the power outage, I took the opportunity to try having the timing system trigger on the rising edge of the 1 PPS signal from the GPS receiver (rather than the falling edge, as it had been doing since before the CDS upgrade). This was done in order to eliminate a timing offset between the clock used by the ADCs and the IRIG-B timestamps from the Spectracom boards. It was successful in that regard, and cleared the timing errors seen in the IOP statewords on most of the front ends.

The two exceptions were c1lsc and c1sus, where something is broken with the DuoTone readbacks. The attached plot shows the DuoTone signal in the last channel of ADC 0 on c1x01 (working normally) vs c1x02 and c1x04 (busted).

However, this change apparently also had some unexplained effect on the way the c1sbr model runs. There were frequent CPU time overruns and IPC errors. So on Sunday afternoon I reverted to the falling-edge trigger. This caused the timing errors to return, but allowed c1sbr to run normally.

Located in /opt/rtcds/caltech/c1/userapps/release/cds/common/scripts

Script 1: Diagreset.sh
Hits the diag reset buttons on the models on c1lsc and c1sus computers.

Script 2: Burtrest.sh
Restores the burt files from "today" 4am. Use a text editor if you want to change the times.

[Jenne, Kyung Ha]

We successfully suspended the 4 eddy current dampers for the first Tip Tilt.  We had some lessons learned, including how to carefully get an allen wrench in between the dampers to do up some of the screws, and how to be careful not to bend the wire while tightening the screws.  More tomorrow...

 We also wrote down the serial numbers (top center of each TT, inscribed by hand) for what tip tilt is installed where.  I then went through the elog to determine which TT was suspended with what kind of wire (thick or thin).  Summary: all installed tip tilts have thick wire, 0.0036" diameter.

As noted in elog 3295, we had found that there was similar hysteresis whether we used the thick or the thin wire, so we had decided not to go back and re-suspend every optic.

Also, since we will redo the pitch balance tomorrow with the new hardware tomorrow, I think we should put in the new LaserOptik mirrors at the same time.  We have not yet gotten phase maps of them, but we might as well do this rebalancing once, rather than twice.

Jamie and I spent some time with tip tilt SN001 this afternoon.  This was installed as SR3, so I was going to put a new LaserOptik mirror in there.  I accidentally snapped one of the wires (I forgot how strong the magnets are - one zipped from the mirror holder and captured the wire).  Jamie and I put the new LaserOptik mirror in, with the wedge correct, but we need to re-resuspend it with the 0.0036" wire tomorrow.  We'll also keep working on re-pitch aligning the other optics.

PR2 needs to be put back as a G&H, and we need to put a LaserOptik mirror into PR3.

 We resuspended SN001 this morning with 0.0036" wire.  We did as Koji suggested, and flipped the wire clamp so the suspension point is a little higher, so we'll see if that helps.  We put LaserOptik mirror SN1 into this TT001.

We put the G&H mirror back into TT004, which is PR2.  We also put a LaserOptik mirror (SN5) into TT005, which is SR3.

Jamie is working on re-pitch aligning TT004 and TT005 (we already did 001), then we can re-install them in the vacuum system later this afternoon.

 The tip tilts have all been pitch-adjusted now, and they have all been put back onto the tables, with the same serial numbers in the same places as we took them out.  Jamie also re-leveled the BS table.

Raji and I will align things after I finish measuring the MC spot positions.

I have made a preliminary sketch of the cabling involved in connecting the Tip-tilt coil drivers.   This is a preliminary document. 

2 questions (so far) regarding your diagram / doc:

We are using 3 of the feed-throughs on the BS chamber, and 1 on the OMC chamber, even though we have 2 TTs on the BS table, 1 on the OMC table, and 1 on the IMC table? Just wanted to check.

Does your list / table at the bottom include all of the cables we already have, as well as the ones we need?  (Or maybe we just have nothing so far, so this is a moot question).

 The scheme currently is to run a 25pin Kapton strip cable from BS to IMC table inside the chamber.  However we cannot do the same for the OMC table since it will cross the bellows which we often remove.  So we need to supply the one tip-tilt on the OMC table from outside.  I could not spot a spare unused feedthough on the OMC chamber.  We will have to swap one of the blank flanges for one with a few feed throughs.

We do not have any of the cables.   So everything listed has to be arranged for.   The pics are from the existing coil driver system on the SUS machine.

Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.

Recent History

The lower blades which I had given to the Physics Workshop for making a vacuum relief hole (using a sinker-EDM process) came back about ten days ago.   Merih Eken <meken@caltech.edu>,  the supervisor at the Physics Dept workshop, handled this matter for us.  The blades were sent to a local EDM machineshop and returned in about three working days ( a weekend intervened). 

Bob cleaned and handed them over to me yesterday evening.  

Current status

Today I have reassembled the four tip-tilts.  I have repacked them in clean bags (double bagged) shifted them to Clean Optics Cabinet (near the ETMX chamber).  The four tip-tilts are in the bottom-most shelf in the cabinet.  There are also some tip-tilt spares in a separate envelope.

Note:  The mirror holder is now held tightly by the eddy current dampers.  This was done for safety of the wires during transportation from LHO.  The eddy current damper in the front of the mirror has to be retracted to allow the mirror holder to swing free.  It has be to about 1mm away from the suspended mirror holder

Work Remaining

1) We need to install the quadrapus cables.  The connector placement on the BOSEM side will have some issues.  It is best to loosen the BOSEM seating as well as the connector seating screws and then push the cable connector into place.  Caution:  when the connector seating screws on the BOSEM are loosened the flexible ckt could be damaged by the loose connector.

2) Insert the mirrors into the mirror holders and balance the suspension such that the mirror's hang vertical and do not have a large yaw offset.

3) Adjust the wire suspension point height so that the flags are in the center of the BOSEM aperture.  Else they will strike against the

4) We need to adjust the position of the BOSEMs such that the shadow sensor signals are at 50%.  This ensures that all the magnets hang at an appropriate distance from their respective coils.

5) To do (3) we need to set up a shadow sensor read-out set-up for one tip-tilt (four sensors)

Final Summary of changes to mirror holder in Tip-Tilt holder.

Determining minimum range for Side Clamp:

1. The initial distance b/w wire-release point and mirror assembly COM = 0.265 mm

2. But this distance is assuming that wire-release point is at mid-point of clamp. So I'm settling on a range of +/- 1mm. The screenshots below confirm range of ~1mm between (1) side screw & protrusion and (2) clamp screw and clamp.

Determining length of tilt-weight assembly rod at the bottom to get 20mRad range

The tilt-weight assembly is made from following Mcmaster parts:
Rod   - 95412A864 18-8 SS  #2-56 Threaded Rod
Nuts  - 91855A103 18-8 SS #2-56 Acorn Cap Nut

Since the weights are fixed, only rod length can be changed to get the angle range.

So a range of 1 mm between nut's inner face and mirror-holder face should be enough. Since holder is 12 mm thick, rod length = 12mm + 2 x 1mm + 2 x (nut length) = 12 + 2 + 9.6 = 23.6 mm = 0.93 inch. So a 1" rod from Mcmaster will be fine.

Here are some tasks that I want someone to work on during my absence.

1. Y-arm alignment for IR

 Basically we gradually have to move onto the Y-arm locking at some point.

Prior to it we need to align the Y arm for IR. Probably we have to touch PZT1 and PZT2.

It would be very nice if the X-arm alignment also gets improved together with this work. 

2. Temperature feedback with a digital control for X end PDH lock

  Need a temperature feedback not with an analog way but with a digital way because we want to put an offset and the feedback signal at the same time (#4198).

 Right now the temperature control input of the laser is connected to a slow DAC (#3850).

Probably we should plug the feedback signal from the PDH box to the fast ADC (i.e. c1iscex), and then connect a fast DAC to the laser temperature.

This entry maybe helpful.

3. Calibration of optical gain for IR arm locking

 In order to evaluate the performance of the green locking, one of the key points is the IR PDH signal.

Because it tells us how much the motion of the X arm is suppressed at IR when the green lock is engaged.

To get this information in m/sqrtHz, we need to know the optical gain.

4. MC servo characterization and PSL frequency noise measurement

 SInce the green beat note tells us the frequency difference between the MC and the arm in the current configuration, we should know how the MC servo is.

Along with this work, I need someone to measure the PSL frequency noise, when it is locked to the MC over the frequency range from 0.01Hz to 1kHz.

5. PLL characterization 

 Solve this issue (#4195) and make it reliable.

1) Fixup REFL165: remove ND filters, get box for PD, dump diode reflections, put less light on diode, change DC transimpedance (?), max power dissipation on BBPD < 0.5 W w/ 25 V bias. Perhaps replace OP27 with TLE2027.

2) Make plan for fixing fiber layout up and down the arms. Need tubing for the whole run. Don't make it cheesy. Two fibers per arm.

3) Fix LSC model to allow user switching of whitening. Get back to working on AutoLock scripts (not Guardian).

4) Manasa, Q, Jenne, tune Oplev servos Tuesday morning/afternoon.

5) Reconnect the other seismometers (Steve, Jenne). For real.

6) Balance PRMI coils at high frequency.

This is just a listing of  CDS problems I still notice today:

1. MC2-MCL button was left ON due to BURT failure. This, of course, screws up our Green locking investigations because of the unintended feedback. Please fix the BURT/button issue.

2. The GCV - FB0 status is RED. I guess this means there's something wrong? Its really a bad idea to have a bunch of whited out or falsely red indicators. No one will ever use these or trust these in the future.
3. MC1/2/3 Lockins are all white. Also, the MODE switches for the dewhitening are all white.
4. Is the MC SIDE coil dewhitening filter synced with anything? It doesn't seem to switch anything. Maybe the dewhite indicators at the top right of the SUS screens can be made to show the state of the binary output instead of just the digital filter
5. MC WFS is all still broken. We need a volunteer to take this on - align beams, replace diodes, fix code/screens.

  I am concentrating on the RF system just now and will be attending to the RF PDs one by one.  Also plan to work on some of the simpler CDS problems when I overlap with Joe.  Will be available for helping out with the beam alignment.

1. ETMX cannot load his filter coefficients. Even if I change the file, the load button doesn't work. I tried changing the lockin filters but they won't load.
2. The ETMX filter modules appear to have 2048 for some of the modules and 16384 for some of the others. How come the make script doesn't make them all 16384?
3. There are a bunch of kill/start scripts in the scripts/ directory instead of in the scripts/FE/ directory. Did this get reverted after a new code exchange?
4. I tried restarting the code using c1startscx, but that doesn't work correctly. It cannot find the burtrb and burtwb files even though they are in the normal path.
5. Kiwamu was using a bunch of cockamamie filters I found.
6. I can't get any minute trend data. I tried on rossa, rosalba, and op440m.

inline cdsToggle /opt/rtcds/caltech/c1/userapps/release/cds/common/src/cdsToggle.c

void cdsToggle(double *in, int inSize, double *out, int outSize){
  static double x = 0;
  static double y = 0;

  if (*in != y){
    y = *in;
    if (y == 1){
      x = (x == 1) ? 0 : 1;
      *out = x;
    }
  }
}

The toilet tank in the big bathroom stopped refilling. I contacted PPService@caltech.edu and put up an "Out of Order sign".

a plumber came in yesterday and fixed the issue.

 Required arm length = 37.7974 +/- 0.02 [m]

This is a preliminary result of the estimation of the Arm length tolerance.

Figure.2  A sensing matrix of the 40-m DRFPMI while changing the position of ETMX/Y by \sigma = 2 cm.

wow. This Monte-Carlo matrix is one of the most advanced optical modeling things I have ever seen. We never had this for any of the interferometers before.

Keiko, Kiwamu

We noticed that we have used wrong code for MICH degree of freedom for both of the ELOG entries on this topic (cavity lengths tolerance search). It will be modified and posted soon.

 Keiko, Kiwamu

Length tolerance of the vertex part is about 5 mm.

Sorry for my procrastinating update on this topic. In my last post, I reported that the length tolerance of the vertex ifo would be 2mm, based on Kiwamu's code on CVS. Then we noticed that the MICH degrees of freedom was wrong in the code. I modified the code and ran again. You can find the modified codes on CVS (40m folder, analyzeDRMITolerance3f.m and DRMITolerance.m)

In this code, the arm lengths were kept to be ideal while some length offsets of random gaussian distribution were added on PRCL, SRCL and MICH lengths. The iteration was 1000 times for each sigma of the random gaussian distribution. The resulting sensing matrix is shown as histogram. Also, a histogram of the demodulation phase separation between MICH and SRCL is plotted by this code, as these two length degrees of freedom will be obtained by one channel separated by the demodulation phase. We check this separation because you want to make sure that the random length offsets does not make the separation of these two signals close.

The result is a bit different from the previous post, in the better way! The length tolerance is about 5 mm for the vertex ifo. Fig.1 shows the sensing matrix. Although signal levels are changed by the random offsets, only few orders of magnitude is changed in each degrees of freedom. Fig.2 shows that the signal separation between MICH and SRCL at  POP55 varies from  55 to 120 degrees, which may be OK. If you have 1cm sigma, it varies from 50 degrees to 150 degrees.

Fig. 1 Histgram of the sensing matrix including 3f channels, when sigma is 5mm. Please note that the x-axis is in long 10.

 Fig. 2 Histogram of the demodulation phase difference between MICH and SRCL, when sigma is 5 mm. To obtain the two signals independently, 90 is ideal. With the random offsets, the demodulation phase difference varies from 55 degrees to 120 degrees.

My next step is to run the similar code for LLO. 

I wanted to check the script archive to see some old settings. I found that the script archive inflated to huge volume (~1TB).
The size of the common NFS volume (/cvs/cds) is 3TB. So it is really significant.

- The scripts living in /opt/rtcds/caltech/c1/scripts are archived daily in /cvs/cds/caltech/scripts_archive as bz2 files. This is done by crontab of megatron (see https://wiki-40m.ligo.caltech.edu/Computers_and_Scripts/CRON)

- In fact, the script folder (say old script folder) /opt/rtcds/caltech/c1/scripts has the size of 10GB. And we have the compressed copy of thi s everyday.

- This large script folder is due to a couple of huge files/folders listed below

• (scripts)/MEDMtab is 5.3GB / This is probably related to the web MEDM view (on nodus) but I don't think the web page is not updated. (i.e. the images are unused)
• (scripts)/MC/logs/AutoLocker.log 2.9GB / This is just the accumulated MC autolocker log.
• (scripts)/GigE 780M / This does not look like scripts but source and object files
• (scripts)/Admin/n2Check.log 224M / This is important but increases every minute.
• (scripts)/ZI 316MB / Zurich Instrument installation. This should not be here.

Here I propose some changes.
For the script archive

• We can remove most of the scripts for the past (say ~2019). We leave an archive file per month.
• For the scripts in 2020, we leave a weekly archive.
• For 2021 and 2022, we leave all the archive files.

For the existing large files/folders

• MEDMtab: the stored files are redundant with the burt snapshots. Remove the image files. Also, we want to move the image-saving location.
• Autolocker.log: simply zap it
• n2Check.log: we should move the saving location
• GigE /ZI: they need a new home where the daily copy is not taken.

As a result of the following work, the file volume of /cvs/cds was reduced from 3.2TB to 2.2TB, and /opt/rtcds/caltech/c1/scripts was reduced from 10GB to 1.5GB

/cvs/cds/caltech/scripts_archive was cleaned up. Now the archive files are reduced to have:

• every month 1st day from 2005 to 2018/12
• every ten days (1, 11, 21) for 2019 and 2020
• everyday for 2021 and 2022

(scripts)/MEDMtab/image was deleted. I can be restore back from one of the script_archive files.

(scripts)/MC/logs/AutoLocker.log was just deleted and refreshed. For the past settings, we can refer autoburt snapshots or script_archive files.

(scripts)/Admin/n2Check.log

• It turned out that the frequency of the check was reduced to once per 10min on Sep 9th, 2021 (unelogged activity).
• The volume of the text since then was not much volume. So I deleted the lines before this date. And the file size is <7MB now.

(scripts)/ZI was moved to /cvs/cds/apps

/opt/rtcds/caltech/c1/burt/autoburt/snapshots

• 2018, 2019, 2020 snapshots were archived in tar.gz.
• These snapshots were then deleted

I noticed (while relocking the MC after Jamie and I zeroed the LSC offsets) that the MC refl power was 4.8 V.  Usually we should be ~4.2, so I closed the PSL shutter and went in to measure the power.  We were injecting ~125mW or a little more.  I had adjusted the power the other day, and through yesterday, it looked fine, but overnight it looks like it drifted up.