40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 240 of 341  Not logged in ELOG logo
ID Date Author Type Category Subjectup
  16283   Thu Aug 19 03:23:00 2021 AnchalUpdateCDSTime synchornization not running

I tried to read a bit and understand the NTP synchronization implementation in FE computers. I'm quite sure that NTP synchronization should be 'yes' if timesyncd are running correctly in the output of timedatectl in these computers. As Koji reported in 15791, this is not the case. I logged into c1lsc, c1sus and c1ioo and saw that RTC has drifted from the software clocks too which does not happen if NTP synchronization was active. This would mean that almost certainly, if the computers are rebooted, the synchronization will be lost and the models will fail to come online.

My current findings are the following (this should be documented in wiki once we setup everything):

  • nodus is running a NTP server using chronyd. One can check the configuration of this NTP serer in /etc/chornyd.conf
  • fb1 is running an NTP server using ntpd that follows nodus and an IP address 131.215.239.14. This can be seen in /etc/ntp.conf.
  • There are no comments to describe what this other server (131.215.239.14) is. Does the GC network have an NTP server too?
  • c1lsc, c1sus and c1ioo all have systemd-timesyncd.service running with configuration file in /etc/systemd/timesyncd.conf.
  • The configuration file set Servers=ntpserver but echo $ntpserver produces nothing (blank) on these computers and I've been unable to find anyplace where ntpserver is defined.
  • In chiara (our name server), the name server file /etc/hosts does not have any entry for ntpserver either.
  • I think the problem might be that these computers are unable to find the ntpserver as it is not defined anywhere.

The solution to this issue could be as simple as just defining ntpserver in the name server list. But I'm not sure if my understanding of this issue is correct. Comments/suggestions are welcome for future steps.

 

  16284   Thu Aug 19 14:14:49 2021 KojiUpdateCDSTime synchornization not running

131.215.239.14 looks like Caltech's NTP server (ntp-02.caltech.edu)
https://webmagellan.com/explore/caltech.edu/28415b58-837f-4b46-a134-54f4b81bee53

I can't say it is correct or not as I did not make the survey at your level. I think you need a few tests of reconfiguring and restarting the NTP clients to see if time synchronization starts. Because the local time is not regulated right now anyway, this operation is safe I think.

 

  16285   Fri Aug 20 00:28:55 2021 AnchalUpdateCDSTime synchornization not running

I added ntpserver as a known host name for address 192.168.113.201 (fb1's address where ntp server is running) in the martian host list in the following files in Chiara:

/var/lib/bind/martian.hosts
/var/lib/bind/rev.113.168.192.in-addr.arpa

Note: a host name called ntp was already defined at 192.168.113.11 but I don't know what computer this is.

Then, I restarted the DNS on chiara by doing:

sudo service bind9 restart

Then I logged into c1lsc and c1ioo and ran following:

controls@c1ioo:~ 0$ sudo systemctl restart systemd-timesyncd.service

controls@c1ioo:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
   Active: active (running) since Fri 2021-08-20 07:24:03 UTC; 53s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 23965 (systemd-timesyn)
   Status: "Idle."
   CGroup: /system.slice/systemd-timesyncd.service
           └─23965 /lib/systemd/systemd-timesyncd

Aug 20 07:24:03 c1ioo systemd[1]: Starting Network Time Synchronization...
Aug 20 07:24:03 c1ioo systemd[1]: Started Network Time Synchronization.
Aug 20 07:24:03 c1ioo systemd-timesyncd[23965]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:24:35 c1ioo systemd-timesyncd[23965]: Using NTP server 192.168.113.201:123 (ntpserver).
controls@c1ioo:~ 0$ timedatectl
      Local time: Fri 2021-08-20 07:25:28 UTC
  Universal time: Fri 2021-08-20 07:25:28 UTC
        RTC time: Fri 2021-08-20 07:25:31
       Time zone: Etc/UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

The same output is shown in c1lsc too. The NTP synchronized flag in output of timedatectl command did not change to yes and the RTC is still 3 seconds ahead of the local clock.

Then I went to c1sus to see what was the status output before rstarting the timesyncd service. I got folloing output:

controls@c1sus:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
   Active: active (running) since Tue 2021-08-17 04:38:03 UTC; 3 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 243 (systemd-timesyn)
   Status: "Idle."
   CGroup: /system.slice/systemd-timesyncd.service
           └─243 /lib/systemd/systemd-timesyncd

Aug 20 02:02:18 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 02:36:27 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:10:35 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:44:43 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:18:51 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:53:00 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 05:27:08 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:01:16 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:35:24 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:09:33 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).

This actually shows that the service was able to find ntpserver correctly at 192.168.113.201 even before I changed the name server file in chiara. So I'm retracting the changes made to name server. They are probably not required.

The configuration files for timesynd.conf are read only even with sudo. I tried changing permissions but that did not work either. Maybe these files are not correctly configured. The man page of timesyncd  says to use field 'NTP' to give the ntp servers. Our files are using field 'Servers'. But since we are not getting any error message, I don't think this is the issue here.

I'll look more into this problem.

  16286   Fri Aug 20 06:24:18 2021 AnchalUpdateCDSTime synchornization not running

I read on some stack exchange that 'NTP synchornized' indicator turns 'yes' in the output of command timedatectl only when RTC clock has been adjusted at some point. I also read that timesyncd does not do the change if the time difference is too much, roughly more than 3 seconds.

So I logged into all FE machines and ran sudo hwclock -w to synchronize them all to the system clocks and then waited if the timesyncd does any correction on RTC. It did not. A few hours later, I found the RTC clocks drifitng again from the system clocks. So even if the timesynd service is running as it should, it si not performing time correction for whatever reason.

Maybe we should try to use some other service?

Quote:
 

The NTP synchronized flag in output of timedatectl command did not change to yes and the RTC is still 3 seconds ahead of the local clock.

 

  16291   Mon Aug 23 22:51:44 2021 AnchalUpdateGeneralTime synchronization efforts

Related elog thread: 16286


I didn't really achieve anything but I'm listing what I've tried.

  • I know now that the timesyncd isn't working because systemd-timesyncd is known to have issues when running on a read-only file system. In particular, the service does not have privileges to change the clock or drift settings at /run/systemd/clock or /etc/adjtime.
  • The workarounds to these problems are poorly rated/reviews in stack exchange and require me to change the /etc/systmd/timesyncd.conf file but I'm unable to edit this file.
  • I know that Paco was able to change these files earlier as the files are now changed and configured to follow a debian ntp pool server which won't work as the FEs do not have internet access. So the conf file needs to be restored to using ntpserver as the ntp server.
  • From system messages, the ntpserver is recognized by the service as shown in the second part of 16285. I really think the issue is in file permissions. the file /etc/adjtime has never been updated since 2017.
  • I got help from Paco on how to edit files for FE machines. The FE machines directories are exported from fb1:/diskless/root.jessie/
  • I restored the /etc/systmd/timesyncd.conf file to how it as before with just servers=ntpserver line. Restarted timesyncd service on all FEs,I tried a few su the synchronization did not happen.
  • I tried a few suggestions from stackexchange but none of them worked. The only rated solution creates a tmpfs directory outside of read-only filesystem and uses that to run timesyncd. So, in my opinion, timesyncd  would never work in our diskless read-only file system FE machines.
  • One issue in an archlinux discussion ended by the questioner resorting to use opennptd from openBSD distribution. The user claimed that opennptd is simple enough that it can run ntp synchornization on a read-only file system.
  • Somehwat painfully, I 'kind of' installed the openntpd tool in the fb1:/diskless/root.jessie directory following directions from here. I had to manually add user group and group for the FEs (which I might not have done correctly). I was not able to get the openntpd daemon to start properly after soe tries.
  • I restored everything back to how it was and restarted timesyncd in c1sus even though it would not do anything really.
Quote:

This time no matter how we try to set the time, the IOPs do not run with "DC status" green. (We kept having 0x4000)

 

  16293   Tue Aug 24 18:11:27 2021 PacoUpdateGeneralTime synchronization not really working

tl;dr: NTP servers and clients were never synchronized, are not synchronizing even with ntp... nodus is synchronized but uses chronyd; should we use chronyd everywhere?


Spent some time investigating the ntp synchronization. In the morning, after Anchal set up all the ntp servers / FE clients I tried restarting the rts IOPs with no success. Later, with Tega we tried the usual manual matching of the date between c1iscex and fb1 machines but we iterated over different n-second offsets from -10 to +10, also without success.

This afternoon, I tried debugging the FE and fb1 timing differences. For this I inspected the ntp configuration file under /etc/ntp.conf in both the fb1 and /diskless/root.jessie/etc/ntp.conf (for the FE machines) and tried different combinations with and without nodus, with and without restrict lines, all while looking at the output of sudo journalctl -f on c1iscey. Everytime I changed the ntp config file, I restarted the service using sudo systemctl restart ntp.service . Looking through some online forums, people suggested basic pinging to see if the ntp servers were up (and broadcasting their times over the local network) but this failed to run (read-only filesystem) so I went into fb1, and ran sudo chroot /diskless/root.jessie/ /bin/bash to allow me to change file permissions. The test was first done with /bin/ping which couldn't even open a socket (root access needed) by running chmod 4755 /bin/ping then ssh-ing into c1iscey and pinging the fb1 machine successfully. After this, I ran chmod 4755 /usr/sbin/ntpd so that the ntp daemon would have no problem in reaching the server in case this was blocking the synchronization. I exited the chroot shell and the ntp daemon in c1iscey; but the ntpstat still showed unsynchronised status. I also learned that when running an ntp query with ntpq -p if a client has succeeded in synchronizing its time to the server time, an asterisk should be appended at the end. This was not the case in any FE machine... and looking at fb1, this was also not true. Although the fb1 peers are correctly listed as nodus, the caltech ntp server, and a broadcast (.BCST.) server from local time (meant to serve the FE machines), none appears to have synchronized... Going one level further, in nodus I checked the time synchronization servers by running chronyc sources the output shows

controls@nodus|~> chronyc sources
210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* testntp1.superonline.net      1  10   377   280  +1511us[+1403us] +/-   92ms
^+ 38.229.59.9                   2  10   377   206  +8219us[+8219us] +/-  117ms
^+ tms04.deltatelesystems.ru     2  10   377   23m    -17ms[  -17ms] +/-  183ms
^+ ntp.gnc.am                    3  10   377   914  -8294us[-8401us] +/-  168ms

I then ran chronyc clients to find if fb1 was listed (as I would have expected) but the output shows this --

Hostname                   Client    Peer CmdAuth CmdNorm  CmdBad  LstN  LstC
=========================  ======  ======  ======  ======  ======  ====  ====
501 Not authorised

So clearly chronyd succeeded in synchronizing nodus' time to whatever server it was pointed at but downstream from there, neither the fb1 or any FE machines seem to be synchronizing properly. It may be as simple as figuring out the correct ntp configuration file, or switching to chronyd for all machines (for the sake of homogeneity?)

  16295   Tue Aug 24 22:37:40 2021 AnchalUpdateGeneralTime synchronization not really working

I attempted to install chrony and run it on one of the FE machines. It didn't work and in doing so, I lost the working NTP client service on the FE computers as well. Following are some details:

  • I added the following two mirrors in the apt source list of root.jessie at /etc/apt/sources.list
    deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
    deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free
  • Then I installed chrony in the root.jessie using
    sudo apt-get install chrony
    • I was getting an error E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory) . To fix this, I had to run:
      sudo mount -t devpts none "$rootpath/dev/pts" -o ptmxmode=0666,newinstance
      sudo ln -fs "pts/ptmx" "$rootpath/dev/ptmx"
    • Then, I had another error to resolve.
      Failed to read /proc/cmdline. Ignoring: No such file or directory
      start-stop-daemon: nothing in /proc - not mounted?
      To fix this, I had to exit to fb1 and run:
      sudo mount --bind /proc /diskless/root.jessie/proc
    • With these steps, chrony was finally installed, but I immediately saw an error message saying:
      Starting /usr/sbin/chronyd...
      Could not open NTP sockets
  • I figured this must be due to ntp running in the FE machines.  I logged into c1iscex and stopped and disabled the ntp service:
    sudo systemctl stop ntp
    sudo systemctl disable ntp
    • I saw some error messages from the above coomand as FEs are read only file systems:
      Synchronizing state for ntp.service with sysvinit using update-rc.d...
      Executing /usr/sbin/update-rc.d ntp defaults
      insserv: fopen(.depend.stop): Read-only file system
      Executing /usr/sbin/update-rc.d ntp disable
      update-rc.d: error: Read-only file system
    • So I went back to chroot in fb1 and ran the two command sabove that failed:
      /usr/sbin/update-rc.d ntp defaults
      /usr/sbin/update-rc.d ntp disable
    • The last line gave the output:
      insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
      insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
    • I igored this and moved forward.
  • I copied the chronyd.service from nodus to the chroot in fb1 and configured it to use nodus as the server. The I started the chronyd.service

    sudo systemctl status chronyd.service
    but got the saem issue of NTP sockets.

    â—Â chronyd.service - NTP client/server
       Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
       Active: failed (Result: exit-code) since Tue 2021-08-24 21:52:30 PDT; 5s ago
      Process: 790 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=1/FAILURE)

    Aug 24 21:52:29 c1iscex systemd[1]: Starting NTP client/server...
    Aug 24 21:52:30 c1iscex chronyd[790]: Could not open NTP sockets
    Aug 24 21:52:30 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
    Aug 24 21:52:30 c1iscex systemd[1]: Failed to start NTP client/server.
    Aug 24 21:52:30 c1iscex systemd[1]: Unit chronyd.service entered failed state.

  • I tried a few things to resolve this, but couldn't get it to work. So I gave up on using chrony and decided to go back to ntp service atleast.

  • I stopped, disabled and checked status of chrony:
    sudo systemctl stop chronyd
    sudo systemctl disable chronyd
    sudo systemctl status chronyd
    This gave the output:

    â—Â chronyd.service - NTP client/server
       Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
       Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:07 PDT; 25s ago

    Aug 24 22:09:07 c1iscex systemd[1]: Starting NTP client/server...
    Aug 24 22:09:07 c1iscex chronyd[2490]: Could not open NTP sockets
    Aug 24 22:09:07 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
    Aug 24 22:09:07 c1iscex systemd[1]: Failed to start NTP client/server.
    Aug 24 22:09:07 c1iscex systemd[1]: Unit chronyd.service entered failed state.
    Aug 24 22:09:15 c1iscex systemd[1]: Stopped NTP client/server.

  • I went back to fb1 chroot and removed chrony package and deleted the configuration files and systemd service files:
    sudo apt-get remove chrony

  • But when I started ntp daemon service back in c1iscex, it gave error:
    sudo systemctl restart ntp
    Job for ntp.service failed. See 'systemctl status ntp.service' and 'journalctl -xn' for details.

  • Status shows:

    sudo systemctl status ntp
    â—Â ntp.service - LSB: Start NTP daemon
       Loaded: loaded (/etc/init.d/ntp)
       Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:56 PDT; 9s ago
      Process: 2597 ExecStart=/etc/init.d/ntp start (code=exited, status=5)

    Aug 24 22:09:55 c1iscex systemd[1]: Starting LSB: Start NTP daemon...
    Aug 24 22:09:56 c1iscex systemd[1]: ntp.service: control process exited, code=exited status=5
    Aug 24 22:09:56 c1iscex systemd[1]: Failed to start LSB: Start NTP daemon.
    Aug 24 22:09:56 c1iscex systemd[1]: Unit ntp.service entered failed state.

  • I tried to enable back the ntp service by sudo systemctl enable ntp. I got similar error messages of read only filesystem as earlier.
    Synchronizing state for ntp.service with sysvinit using update-rc.d...
    Executing /usr/sbin/update-rc.d ntp defaults
    insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
    insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
    insserv: fopen(.depend.stop): Read-only file system
    Executing /usr/sbin/update-rc.d ntp enable
    update-rc.d: error: Read-only file system

    • I went back to chroot in fb1 and ran:
      /usr/sbin/update-rc.d ntp defaults
      insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
      insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
      and
      /usr/sbin/update-rc.d ntp enable

  • I came back to c1iscex and tried restarting the ntp service but got same error messages as above with exit code 5.

  • I checked c1sus, the ntp was running there. I tested the configuration by restarting the ntp service, and then it failed with same error message. So the remaining three FEs, c1lsc, c1ioo and c1iscey have running ntp service, but they won't be able to restart.

  • As a last try, I rebooted c1iscex to see if ntp comes back online nicely, but it doesn't.

Bottom line, I went to try chrony in the FEs, and I ended up breaking the ntp client services on the computers as well. We have no NTP synchronization in any of the FEs.

Even though Paco and I are learning about the ntp and cds stuff, I think it's time we get help from someone with real experience. The lab is not in a good state for far too long.

Quote:

tl;dr: NTP servers and clients were never synchronized, are not synchronizing even with ntp... nodus is synchronized but uses chronyd; should we use chronyd everywhere?

 

  16292   Tue Aug 24 09:22:48 2021 AnchalUpdateGeneralTime synchronization working now

Jamie told me to use chroot to log in into the chroot jail of debian os that are exported for the FEs and install ntp there. I took following steps at the end of which, all FEs have NTP synchronized now.

  • I logged into fb1 through nodus.
  • chroot /diskless/root.jessie /bin/bash took me to the bash terminal for debian os that is exported to all FEs.
  • Here, I ran sudo apt-get install ntp which ran without any errors.
  • I then edited the file in /etc/ntp.conf , i removed the default servers and added following lines for servers (fb1 and nodus ip addresses):
    server 192.113.168.201
    server 192.113.168.201
  • I logged into each FE machine and ran following commands:
    sudo systemctl stop systemd-timesyncd.service; sudo systemctl status systemd-timesyncd.service;
    timedatectl; sleep 2;sudo systemctl daemon-reload;  sudo systemctl start ntp; sleep 2; sudo systemctl status ntp; timedatectl
    sudo hwclock -s
    • The first line ensures that systemd-timesyncd.service is not running anymore. I did not uninstall timesyncd and left its configuration file as it is.
    • The second line first shows the times of local and RTC clocks. Then reloads the daemon services to get ntp registered. Then starts ntp.service and shows it's status. Finally, the timedatectl command shows the synchronized clocks and that NTP synchronization has occured.
    • The last line sets the local clock same as RTC clock. Even though this wasn't required as I saw that the clocks were already same to seconds, I just wanted a point where all the local clocks are synchronized to the ntp server.
  • Hopefully, this would resolve our issue of restarting the models anytime some glitch happens or when we need ot update something in one of them.

Edit Tue Aug 24 10:19:11 2021:

I also disabled timesyncd on all FEs using sudo systemctl disable systemd-timesyncd.service

I've added this wiki page for summarizing the NTP synchronization knowledge.

  8098   Mon Feb 18 11:54:15 2013 Max HortonUpdateSummary PagesTiming Issues and Calendars

Crontab: The bug of data only plotting until 5PM is being investigated.  The crontab's final call to the summary page generator was at 5PM.  This means that the data plots were not being generated after 5PM, so clearly they never contained data from after 5PM.  In fact, the original crontab reads:

0 11,5,11,17 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1

I'm not exactly sure what inspired these entries.  The 11,5,11,17 entries are supposed to be the hours at which the program is run.  Why is it run twice at 11?  I assume it was just a typo or something.

The final call time was changed to 11:59PM in an attempt to plot the entire day's data, but this method didn't appear to work because the program would still be running past midnight, which was apparently inhibiting its functionality (most likely, the day change was affecting how the data is fetched).  The best solution is probably to just wait until the next day, then call the summary page generator on the previous day's data.  This will be implemented soon.

Calendars: Although the calendar tabs on the left side of the page were fixed, the calendars displayed at: https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/ appear to still have squished together text.  The calendar is being fetched from https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/calendar.html and displayed in the page.  This error is peculiar because the URL from which the calendar is being fetched does NOT have squished together text, but the resulting calendar at 40m-summary/calendar/ will not display spaces between the text.  This issue is still being investigated.

  10193   Mon Jul 14 13:03:23 2014 AkhilSummaryElectronicsTiming Issues of Mini Circuits UFC-6000: Solved

Main Problem:

The frequency counter (FC) takes in an analog RF input(signal) and outputs the frequency of the signal(Ranging from 1 MHz- 6000 MHz) in the digital domain (into a processor). The FC samples the data with a given sample rate( user defined) which ranges from 0.1 s to 1 s(faced problems in fixing this initially).  For data acquisition, we have been using a Raspberry Pi(as a processor) which is connected to the martian network and can communicate with the computers inside the 40m.  The ultimate challenge which I faced( and been knocking my head off from past two-three weeks) is the synchronization of clocks between the Raspberry Pi and the FC i.e the clock which the FC uses to sample and dump data( every 'x' s) and the clock inside the raspberry pi( used  in the loop to wait for a particular amount of time the frequency counter takes to dump successive data).

 

Steps Taken:

  • To address this problem, first I added an external clock circuit which monitors the Raspberry Pi and the FC to dump and read data at a particular rate(which is equal to the sampling rate of the FC)In detail: http://nodus.ligo.caltech.edu:8080/40m/10129. 
  • While doing so, at first the level trigger algorithm was used which means that the external clock frequency was half as that of  the reciprocal of the sampling rate and a trigger was seen every time the level shifts from +DC to -DC(of the external square wave).
  • But this did not completely mitigate the issues and there were still few issues on how quickly the ADC reads the signal and R Pi processes it.
  • To minimize these issues completely, an edge trigger algorithm which detects a pos edge(rising)  of the clock was used. The clock  frequency is now equal to the reciprocal of the sampling rate. This algorithm showed better results and greatly minimized the drift of the sampling time.

Psuedo Code(code attached):

open device : FC via USB-HID;

open device : ADC via I2C;

always(for t= recording time):

            read data from ADC(external clock);

            if pos edge detected:

                    read data from FC and store it in a register;

             else read data from ADC;

end

write data stored in the register to a file( can be an Epics channel or a text file);

 

Results:

The attached are the plots showing the time between samples for a large number of samples taken for different sampling times of the FC. The percentage error is the percentage of standard error in the timing between two samples for the data for the entire measurement. It can be inferred that this error has been cut down to the order of ms.

 

To do next:

  • I have started taking phase measurements( analysis and plots will follow this elog) and also the PSD plots with the improved timing characteristics.

 

 

 

             

 

Attachment 1: 0.2timinganalysis.png
0.2timinganalysis.png
Attachment 2: 0.3timinganalysis.png
0.3timinganalysis.png
Attachment 3: 0.5timinganalysis.png
0.5timinganalysis.png
Attachment 4: 1stiminganalysis.png
1stiminganalysis.png
Attachment 5: pdf.zip
  10194   Mon Jul 14 14:28:27 2014 KojiSummaryElectronicsTiming Issues of Mini Circuits UFC-6000: Solved

Looks good. Now you have the internal timer to verify the external clock.
If you can realize the constant rate sampling without employing the external clock, that's quite handy.

  10199   Tue Jul 15 01:31:13 2014 AkhilSummaryElectronicsTiming Issues of Mini Circuits UFC-6000: Solved

The attached are the PSD plots with improved FC timing(with the same code as in http://nodus.ligo.caltech.edu:8080/40m/10151). More plots(Phase and PSD) to follow.

 

 

 

Attachment 1: qnoise.png
qnoise.png
Attachment 2: qnoise.pdf
qnoise.pdf
  1581   Wed May 13 12:41:14 2009 josephbUpdateCamerasTiming and stability tests of GigE Camera code

At the request of people down at LLO I've been trying to work on the reliability and speed of the GigE camera code.  In my testing, after several hours, the code would tend to lock up on the camera end.  It was also reported at LLO after several minutes the camera display would slow down, but I haven't been able to replicate that problem.

I've recently added some additional error checking and have updated to a more recent SDK which seems to help.  Attached are two plots of the frames per second of the code.  In this case, the frames per second  are measured as the time between calls to the C camera code for a new frame for gstreamer to encode and transmit.  The data points in the first graph are actually the averaged time for sets of 1000 frames.  The camera was sending 640x480 pixel frames, with an exposure time of 0.01 seconds.  Since the FPS was mostly between 45 and 55, it is taking the code roughly 0.01 second to process, encode, and transmit a frame.

During the test, the memory usage by the server code was roughly 1% (or 40 megabytes out of 4 gigabytes) and 50% of a CPU (out a total of  CPUs).

Attachment 1: newCodeFPS.png
newCodeFPS.png
Attachment 2: newCodeFPS_hist.png
newCodeFPS_hist.png
  15515   Wed Aug 12 17:36:42 2020 gautamUpdateCDSTiming distribution slot availability

See Attachment #1. J8 was connected to a "LASTI timing slave" sitting in the rack that Chiara lives in - we don't use this for anything and I confirmed that there was no effect on the RTCDS when I pulled that fiber out. The LASTI timing slave also had a blinky that was blinking when the fiber was plugged in - which I take to believe that the slot works. 

Can we get away with just using these two available slots, J8 and J13? Do we really need three new expansion chassis?

Attachment 1: IMG_8706.JPG
IMG_8706.JPG
  15518   Wed Aug 12 20:14:06 2020 KojiUpdateCDSTiming distribution slot availability

I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.

  15521   Thu Aug 13 11:30:19 2020 gautamUpdateCDSTiming distribution slot availability

That's great. I wonder if we can also get away with not adding new Dolphin infrastructure. I'd really like to avoid changing any IPC drivers.

Quote:

I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.

  15522   Thu Aug 13 13:35:13 2020 KojiUpdateCDSTiming distribution slot availability

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

  15525   Fri Aug 14 10:03:37 2020 JonUpdateCDSTiming distribution slot availability

That's great news we won't have to worry about a new timing fanout for the two new machines, c1bhd and c1sus2. And there's no plan to change Dolphin IPC drivers. The plan is only to install the same (older) version of the driver on the two new machines and plug into free slots in the existing switch.

Quote:

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

  4004   Wed Dec 1 13:41:21 2010 josephb, alex, rolfUpdateCDSTiming is back

Problem:

We had timing problems across the front ends.

Solution:

Noticing that the 1PPS reference was not blinking on the Master Timing Distribution box.  It was supposed to be getting a signal from the c0dcu1 VME crate computer, but this was not happening.

We disconnected the timing signal going into c0dcu1, coming from c0daqctrl, and connected the 1PPS directly from c0daqctrl to the Ref In for the Master Timing distribution box (blue box with lots of fibers coming out of it in 1X5).

We now have agreement in timing between front ends.

After several reboots we now have working RFM again, along with computers who agree on the current GPS time along with the frame builder.

Status:

RFM is back and testpoints should be happy.

We still don't have a working binary output for the X end.  I may need to get a replacement backplane with more than 4 slots if the 1st slot of this board has the same problem as the large boards.

I have burt restored the c1ioo, c1mcs, c1rms, c1sus, and c1scx processes, and optics look to be damped.

 

  14386   Fri Jan 4 17:43:24 2019 gautamUpdateCDSTiming issues

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

  • Now, old frame data is no longer being overwritten
  • For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

  • The usual FE/FB power cycling did not fix the problem.
  • The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
  • This was off by 2 years.
  • The way this worked up till now was by adding a fixed offset to this time.
    • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
    • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
    • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

  1. First, we tried changing the offset in the daqdrc.fw file only.
    • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
    • This did not fix the problem.
  2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
    • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
    • Added the following lines:
      /* 2018 had no leap seconds or leap days, so adjust for that */
             pHardware->gpsOffset += 31536000;
    • re-built and installed the modified symmetricom driver.
    • Checked that cat /proc/gps now yields the correct time.
    • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
    • With these steps, the frames were being written to /frames with the correct timestamp.
  3. Next, we checked the timing on the FEs
    • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
    • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
    • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
    • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
    • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Quote:
 
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?
  14392   Wed Jan 9 11:33:35 2019 gautamUpdateCDSTiming issues still persist

Summary:

The gps time mismatch between /proc/gps and gpstime seems to be resolved. However, the 0x4000 DC errors still persist. It is not clear to me why.

Details:

On the phone with J Hanks on Friday, he reminded me that c1sus seems to be the only machine with an IRIG-B timing card installed. I can't find the elog but I remembered that Jamie, ericq and I had done this work in 2016 (?), and I also remembered Jamie saying it wasn't working exactly as expected. Since the DAQ was working fine before this card was installed, and since there are no problems with the recording of channels from the other four FE machines without this card installed, I decided to simply pull out the card from the expansion chassis. The card has been stored in the CDS/FE cabinet along the Y arm for now. There was also a cable that interfaces to the card which brings over the 1pps from the GPS unit, which has also been stored in the CDS/FE cabinet.

This seems to have resolved the mismatch between the gpstime reported by cat /proc/gps and the gpstime commands - Attachment #1 (the <1 second mismatch is presumably due to the deadtime between commands). However, the 0x4000 DC errors still persist. I'll try the full power cycle of FEs and FB which has fixed this kind of error in the past, but apart from that, I'm out of ideas.

Update 1215:

Following the instructions in this elog did not fix the problem. The problem seems to be with the daqd_fw service, which reports the following:

controls@fb1:~ 0$ sudo systemctl status daqd_fw.service 
● daqd_fw.service - Advanced LIGO RTS daqd frame writer
   Loaded: loaded (/etc/systemd/system/daqd_fw.service; enabled)
   Active: failed (Result: start-limit) since Wed 2019-01-09 12:17:12 PST; 2min 0s ago
  Process: 2120 ExecStart=/usr/bin/daqd_fw -c /opt/rtcds/caltech/c1/target/daqd/daqdrc.fw (code=killed, signal=ABRT)
 Main PID: 2120 (code=killed, signal=ABRT)

Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Jan 09 12:17:12 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Jan 09 12:17:12 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
                                                     

Update 1530:

The frame-writer error was tracked down to a C0EDCU issue. Jon told me that the Hornet CC1 pressure gauge channel was renamed to . C1:Vac-CC1_pressure, and I made the change in the C0EDCU file. However, it returns a value of 9990000000.0, which the frame writer is not happy about... Keeping the old channel name makes the frame-writer run again (although the actual data is bunk).

Update 1755:

J Hanks suggested adding a 1 second offset to the daqdrc config files. This has now fixed the 0x4000 errors, and we are back to the "nominal" RTCDS status screen now - Attachment #2.

Attachment 1: gpstimeSync.png
gpstimeSync.png
Attachment 2: Screenshot_from_2019-01-09_17-56-58.png
Screenshot_from_2019-01-09_17-56-58.png
  5731   Mon Oct 24 20:00:21 2011 MirkoUpdateCDSTiny little scripts

Located in /opt/rtcds/caltech/c1/userapps/release/cds/common/scripts

Script 1: Diagreset.sh
Hits the diag reset buttons on the models on c1lsc and c1sus computers.

Script 2: Burtrest.sh
Restores the burt files from "today" 4am. Use a text editor if you want to change the times.

  3162   Tue Jul 6 17:38:27 2010 JenneUpdateSUSTip Tilt Progress!

[Jenne, Kyung Ha]

We successfully suspended the 4 eddy current dampers for the first Tip Tilt.  We had some lessons learned, including how to carefully get an allen wrench in between the dampers to do up some of the screws, and how to be careful not to bend the wire while tightening the screws.  More tomorrow...

  7601   Tue Oct 23 18:12:18 2012 JenneUpdateAlignmentTip tilt wires - the truth

Quote:

At this point I cried foul.  This is not an acceptable situation.  Very little stimulation to the tip-tilts can repoint the beam inside the PR cavity.

Steve says that the TT weights, which will attach to the base of the TT mirror mounts and should help keep the mirrors vertical and not hysteretic, are being baked now and should be available tomorrow. 

 We also wrote down the serial numbers (top center of each TT, inscribed by hand) for what tip tilt is installed where.  I then went through the elog to determine which TT was suspended with what kind of wire (thick or thin).  Summary: all installed tip tilts have thick wire, 0.0036" diameter.

As noted in elog 3295, we had found that there was similar hysteresis whether we used the thick or the thin wire, so we had decided not to go back and re-suspend every optic.

Also, since we will redo the pitch balance tomorrow with the new hardware tomorrow, I think we should put in the new LaserOptik mirrors at the same time.  We have not yet gotten phase maps of them, but we might as well do this rebalancing once, rather than twice.

 

As-installed tip tilt list
Serial number Installed as Wire thickness Notes, elog reference
001 SR 3 0.0036" See elog 3437
002 SR 2 0.0036" See elog 3295
003 PR 2 0.0036" No elog, but inferred since there were 4 with thick wire, and #004 is the thin wire one.  Elog 3437 has notes on the 4 thick, 1 thin situation.
004 spare, dirty originally 0.0017", but looks redone with thicker wire See elog 3295
005 PR 3 0.0036" Was supposed to be spare according to elog 3437, but was installed.  See elog 3437

 

  7625   Thu Oct 25 20:44:11 2012 JenneUpdateSUSTip tilts in progress

Jamie and I spent some time with tip tilt SN001 this afternoon.  This was installed as SR3, so I was going to put a new LaserOptik mirror in there.  I accidentally snapped one of the wires (I forgot how strong the magnets are - one zipped from the mirror holder and captured the wire).  Jamie and I put the new LaserOptik mirror in, with the wedge correct, but we need to re-resuspend it with the 0.0036" wire tomorrow.  We'll also keep working on re-pitch aligning the other optics.

PR2 needs to be put back as a G&H, and we need to put a LaserOptik mirror into PR3.

  7630   Fri Oct 26 10:44:25 2012 JenneUpdateSUSTip tilts in progress

Quote:

Jamie and I spent some time with tip tilt SN001 this afternoon.  This was installed as SR3, so I was going to put a new LaserOptik mirror in there.  I accidentally snapped one of the wires (I forgot how strong the magnets are - one zipped from the mirror holder and captured the wire).  Jamie and I put the new LaserOptik mirror in, with the wedge correct, but we need to re-resuspend it with the 0.0036" wire tomorrow.  We'll also keep working on re-pitch aligning the other optics.

PR2 needs to be put back as a G&H, and we need to put a LaserOptik mirror into PR3.

 We resuspended SN001 this morning with 0.0036" wire.  We did as Koji suggested, and flipped the wire clamp so the suspension point is a little higher, so we'll see if that helps.  We put LaserOptik mirror SN1 into this TT001.

We put the G&H mirror back into TT004, which is PR2.  We also put a LaserOptik mirror (SN5) into TT005, which is SR3.

Jamie is working on re-pitch aligning TT004 and TT005 (we already did 001), then we can re-install them in the vacuum system later this afternoon.

  7631   Fri Oct 26 13:08:14 2012 JenneUpdateSUSTip tilts in progress

Quote:

Quote:
 

Jamie is working on re-pitch aligning TT004 and TT005 (we already did 001), then we can re-install them in the vacuum system later this afternoon.

 The tip tilts have all been pitch-adjusted now, and they have all been put back onto the tables, with the same serial numbers in the same places as we took them out.  Jamie also re-leveled the BS table.

Raji and I will align things after I finish measuring the MC spot positions.

  6756   Tue Jun 5 20:42:59 2012 SureshSummaryIOOTip-Tilt Cabling

I have made a preliminary sketch of the cabling involved in connecting the Tip-tilt coil drivers.   This is a preliminary document. 

40m_Tip-tilt_cabling.png

 

 

  Required parts Quantity Solution
1) DAC Card inserted into C1IOO machine 1 buy or borrow from Cymacs
2) SCSI cable from DAC to D080303 box 1 buy or find at the 40m
3) D080303 box (SCSI to IDC breakout box) 1 Jay may have had spare, if not we have to make one
4) 40 pin IDC cables from D080303 to AntiImaging filter 2 Jay may have kept some stock if not make them
5) 10 pin IDC cables from Anti Imaging filters to Whitening filters 2 make
6) sma to lemo cables from Whitening to coil drivers 4x4=16 buy
7) 15pin IDC to 25pin DSub cables from drivers to feedthroughs on the chambers 4  (length?) make
8) 25pin DSub feedthrough on OMC chamber 1 check in 40m stock else buy
9) 25pin DSub  Kapton strip cable from OMC wall to table top 1 check if any spare are available in aLIGO stock
10) 25pin DSub Kapton strip cable from post to tip-tilt 4 aLIGO team said they have a few to spare if not buy
10) Quadrapus cables on the tip-tilts 4 Jamie is negotiating with aLIGO cable team

 

  6759   Tue Jun 5 22:39:06 2012 JenneSummaryIOOTip-Tilt Cabling

2 questions (so far) regarding your diagram / doc:

We are using 3 of the feed-throughs on the BS chamber, and 1 on the OMC chamber, even though we have 2 TTs on the BS table, 1 on the OMC table, and 1 on the IMC table? Just wanted to check.

Does your list / table at the bottom include all of the cables we already have, as well as the ones we need?  (Or maybe we just have nothing so far, so this is a moot question).

  6762   Wed Jun 6 01:23:32 2012 SureshSummaryIOOTip-Tilt Cabling

Quote:

2 questions (so far) regarding your diagram / doc:

We are using 3 of the feed-throughs on the BS chamber, and 1 on the OMC chamber, even though we have 2 TTs on the BS table, 1 on the OMC table, and 1 on the IMC table? Just wanted to check.

Does your list / table at the bottom include all of the cables we already have, as well as the ones we need?  (Or maybe we just have nothing so far, so this is a moot question).

 The scheme currently is to run a 25pin Kapton strip cable from BS to IMC table inside the chamber.  However we cannot do the same for the OMC table since it will cross the bellows which we often remove.  So we need to supply the one tip-tilt on the OMC table from outside.  I could not spot a spare unused feedthough on the OMC chamber.  We will have to swap one of the blank flanges for one with a few feed throughs.

We do not have any of the cables.   So everything listed has to be arranged for.   The pics are from the existing coil driver system on the SUS machine.

 

  14742   Wed Jul 10 10:04:09 2019 gautamUpdateSUSTip-Tilt moved from South clean cabinet to bake lab cleanroom

Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.

  6770   Wed Jun 6 19:46:46 2012 SureshSummaryIOOTip-tilt assembly: current status and work remaining

 

Recent History

The lower blades which I had given to the Physics Workshop for making a vacuum relief hole (using a sinker-EDM process) came back about ten days ago.   Merih Eken <meken@caltech.edu>,  the supervisor at the Physics Dept workshop, handled this matter for us.  The blades were sent to a local EDM machineshop and returned in about three working days ( a weekend intervened). 

IMG_0685.JPG  IMG_0687.JPG

Bob cleaned and handed them over to me yesterday evening.  

Current status

Today I have reassembled the four tip-tilts.  I have repacked them in clean bags (double bagged) shifted them to Clean Optics Cabinet (near the ETMX chamber).  The four tip-tilts are in the bottom-most shelf in the cabinet.  There are also some tip-tilt spares in a separate envelope.

Note:  The mirror holder is now held tightly by the eddy current dampers.  This was done for safety of the wires during transportation from LHO.  The eddy current damper in the front of the mirror has to be retracted to allow the mirror holder to swing free.  It has be to about 1mm away from the suspended mirror holder

Work Remaining

1) We need to install the quadrapus cables.  The connector placement on the BOSEM side will have some issues.  It is best to loosen the BOSEM seating as well as the connector seating screws and then push the cable connector into place.  Caution:  when the connector seating screws on the BOSEM are loosened the flexible ckt could be damaged by the loose connector.

2) Insert the mirrors into the mirror holders and balance the suspension such that the mirror's hang vertical and do not have a large yaw offset.

3) Adjust the wire suspension point height so that the flags are in the center of the BOSEM aperture.  Else they will strike against the

4) We need to adjust the position of the BOSEMs such that the shadow sensor signals are at 50%.  This ensures that all the magnets hang at an appropriate distance from their respective coils.

5) To do (3) we need to set up a shadow sensor read-out set-up for one tip-tilt (four sensors)

 

Attachment 2: IMG_0687.JPG
IMG_0687.JPG
  14047   Mon Jul 9 17:29:28 2018 Udit KhandelwalSummaryTip-TIltTipTilt mirror holder final changes

Final Summary of changes to mirror holder in Tip-Tilt holder.

Determining minimum range for Side Clamp:

1. The initial distance b/w wire-release point and mirror assembly COM = 0.265 mm


2. But this distance is assuming that wire-release point is at mid-point of clamp. So I'm settling on a range of +/- 1mm. The screenshots below confirm range of ~1mm between (1) side screw & protrusion and (2) clamp screw and clamp.

Determining length of tilt-weight assembly rod at the bottom to get \pm 20mRad range

The tilt-weight assembly is made from following Mcmaster parts:
Rod   - 95412A864 18-8 SS  #2-56 Threaded Rod
Nuts  - 91855A103 18-8 SS #2-56 Acorn Cap Nut

Since the weights are fixed, only rod length can be changed to get the angle range.

tan \theta =\frac{d}{h}

d= h \times tan\theta = 34.25\text{mm} \times tan(20 \text{mRad}) = 0.69 \text{mm}
So a range of 1 mm between nut's inner face and mirror-holder face should be enough. Since holder is 12 mm thick, rod length = 12mm + 2 x 1mm + 2 x (nut length) = 12 + 2 + 9.6 = 23.6 mm = 0.93 inch. So a 1" rod from Mcmaster will be fine.

Attachment 4: 2-1.png
2-1.png
  4199   Tue Jan 25 06:48:55 2011 kiwamuUpdateGreen LockingTo do list

Here are some tasks that I want someone to work on during my absence.

1. Y-arm alignment for IR

 Basically we gradually have to move onto the Y-arm locking at some point.

Prior to it we need to align the Y arm for IR. Probably we have to touch PZT1 and PZT2.

It would be very nice if the X-arm alignment also gets improved together with this work. 

 

2. Temperature feedback with a digital control for X end PDH lock

  Need a temperature feedback not with an analog way but with a digital way because we want to put an offset and the feedback signal at the same time (#4198).

 Right now the temperature control input of the laser is connected to a slow DAC (#3850).

Probably we should plug the feedback signal from the PDH box to the fast ADC (i.e. c1iscex), and then connect a fast DAC to the laser temperature.

This entry maybe helpful.

 

3. Calibration of optical gain for IR arm locking

 In order to evaluate the performance of the green locking, one of the key points is the IR PDH signal.

Because it tells us how much the motion of the X arm is suppressed at IR when the green lock is engaged.

To get this information in m/sqrtHz, we need to know the optical gain.

 

4. MC servo characterization and PSL frequency noise measurement

 SInce the green beat note tells us the frequency difference between the MC and the arm in the current configuration, we should know how the MC servo is.

Along with this work, I need someone to measure the PSL frequency noise, when it is locked to the MC over the frequency range from 0.01Hz to 1kHz.

 

5. PLL characterization 

 Solve this issue (#4195) and make it reliable.

  9667   Mon Feb 24 23:43:10 2014 ranaSummaryGeneralToDo

1) Fixup REFL165: remove ND filters, get box for PD, dump diode reflections, put less light on diode, change DC transimpedance (?), max power dissipation on BBPD < 0.5 W w/ 25 V bias. Perhaps replace OP27 with TLE2027.

2) Make plan for fixing fiber layout up and down the arms. Need tubing for the whole run. Don't make it cheesy. Two fibers per arm.

3) Fix LSC model to allow user switching of whitening. Get back to working on AutoLock scripts (not Guardian).

4) Manasa, Q, Jenne, tune Oplev servos Tuesday morning/afternoon.

5) Reconnect the other seismometers (Steve, Jenne). For real.

6) Balance PRMI coils at high frequency.

  50   Thu Nov 1 19:53:02 2007 Andrey RodionovBureaucracyPhotosTobin's picture
Attachment 1: DSC_0053.JPG
DSC_0053.JPG
  4282   Mon Feb 14 01:19:18 2011 ranaUpdateCDSToday's CDS problems

This is just a listing of  CDS problems I still notice today:

  1. MC2-MCL button was left ON due to BURT failure. This, of course, screws up our Green locking investigations because of the unintended feedback. Please fix the BURT/button issue.

  2. The GCV - FB0 status is RED. I guess this means there's something wrong? Its really a bad idea to have a bunch of whited out or falsely red indicators. No one will ever use these or trust these in the future.
  3. MC1/2/3 Lockins are all white. Also, the MODE switches for the dewhitening are all white.
  4. Is the MC SIDE coil dewhitening filter synced with anything? It doesn't seem to switch anything. Maybe the dewhite indicators at the top right of the SUS screens can be made to show the state of the binary output instead of just the digital filter
  5. MC WFS is all still broken. We need a volunteer to take this on - align beams, replace diodes, fix code/screens.
  4285   Mon Feb 14 07:58:43 2011 SureshUpdateCDSToday's CDS problems

 

  I am concentrating on the RF system just now and will be attending to the RF PDs one by one.  Also plan to work on some of the simpler CDS problems when I overlap with Joe.  Will be available for helping out with the beam alignment.

 

 

  4309   Wed Feb 16 23:54:34 2011 ranaUpdateCDSToday's CDS problems
  1. ETMX cannot load his filter coefficients. Even if I change the file, the load button doesn't work. I tried changing the lockin filters but they won't load.
  2. The ETMX filter modules appear to have 2048 for some of the modules and 16384 for some of the others. How come the make script doesn't make them all 16384?
  3. There are a bunch of kill/start scripts in the scripts/ directory instead of in the scripts/FE/ directory. Did this get reverted after a new code exchange?
  4. I tried restarting the code using c1startscx, but that doesn't work correctly. It cannot find the burtrb and burtwb files even though they are in the normal path.
  5. Kiwamu was using a bunch of cockamamie filters I found.
  6. I can't get any minute trend data. I tried on rossa, rosalba, and op440m.
MC damp dataviewer diaggui AWG c1lsc c1ioo c1sus c1iscex c1iscey RFM The Dolphins Sim.Plant Frame builder TDS Cabling
                             
  5214   Fri Aug 12 17:27:49 2011 YoichiSummaryCDSToggle button for RCG
Bottom line: I made an RCG block to realize a toggle button easily.

Read on if you need such a button, or if you want to know how to
write a new RCG block with C.

-----------------
When I was making MEDM screens for FFC, I wanted to have a toggle
button to enable/disable the FFC path.
I wanted to have something like the ON/OFF buttons of the filter bank
screens, the one changes its state every time I click on it.
However, I could not find an easy way to realize that.

From MEDM, I can send a value to an EPICS channel using a "Message Button".
This value is always the same, say 1.
In the RCG model, I used a cdsEpicsMomentary block so that whenever the channel
gets 1, it stays to be 1 for a while and turns back to 0 in a second or so.
This generates a pulse of 1 when I click on a message button on a MEDM screen.
Then I needed a block to keep its internal state (0 or 1), and flips its state
whenever it receives a pulse of 1.
Since I couldn't find such a block in the current RCG library, I implemented one
using the cdsFunctionCall block. It allows you to implement a block with C code.

There is a good explanation of how to use this block in the CDS_PARTS library.
Here is basically what I did.

(1) Drag and drop thee cdsFunctionCall block to my model.

(2) In the "Block Properties", I put the following line in the Description field.
inline cdsToggle /opt/rtcds/caltech/c1/userapps/release/cds/common/src/cdsToggle.c
This means to call a function cdsToggle(), whose code is in the file indicated above.

(3) The contents of the source code is very simple.
void cdsToggle(double *in, int inSize, double *out, int outSize){
  static double x = 0;
  static double y = 0;

  if (*in != y){
    y = *in;
    if (y == 1){
      x = (x == 1) ? 0 : 1;
      *out = x;
    }
  }
}
The function prototype is always the same. *in and *out are the pointers to the arrays of doubles
for input and output signals of the block. In simuLink, the signals have to be
multiplexed so that the RCG can know how many signals are handed to or returned from the function.
In order to keep the internal state of my custom block, I used "static" keyword in the
declaration of the variables. The rest of the code should be obvious.

(4) Just compile the model as usual. The RCG will automatically include the source code and put
a call to the function in the proper place.

I made the block a library so that people can use it.
/opt/rtcds/caltech/c1/userapps/trunk/cds/common/models/cdsToggle.mdl
is the one.
For the usage of it, please have a look at
/opt/rtcds/caltech/c1/userapps/trunk/isc/c1/models/c1lsc
  16485   Wed Nov 24 17:13:31 2021 YehonathanMetaphysicsGeneralToilet tank broken

The toilet tank in the big bathroom stopped refilling. I contacted PPService@caltech.edu and put up an "Out of Order sign".

  16487   Tue Nov 30 11:03:44 2021 YehonathanMetaphysicsGeneralToilet tank broken

a plumber came in yesterday and fixed the issue.

Quote:

The toilet tank in the big bathroom stopped refilling. I contacted PPService@caltech.edu and put up an "Out of Order sign".

 

  483   Fri May 16 17:27:55 2008 AndreyOmnistructureGeneralToilets are broken, do not use them !!!

Both toilets in 40-meter were constantly flushing, the leaking water was on the floor inside of the restrooms, so

BOTH RESTROOMS ARE CLOSED TILL MONDAY


I have heard the constant loud sound of flushing water, opened the door, and was unpleasantly surprised because all the floor was under the layer of water and the toilets were constantly flushing. I called security at X5000, a plumber came in and told that a team of plumbers needs to repair the flushing system after the weekend. The plumber today just shut off the flushing water, wiped off the floor and told not to use the restrooms in the weekend. We should expect a team of plumbers on Monday.

Sinks are working, so you can wash your hands.
  5076   Sun Jul 31 17:28:34 2011 kiwamuSummaryLSCTolerance of Arm length = 2 cm

 Required arm length = 37.7974 +/- 0.02 [m]

This is a preliminary result of the estimation of the Arm length tolerance.

This number was obtained from a simulation based on Optickle.
Note that the simulation was done by considering misplacements in only the arm lengths while keeping PRCL, SRCL and MICH at the ideal lengths.
Therefore the tolerance will be somewhat tighter if misplacements in the central part are taken into account.

Next : check 3f signals, and include misplacements in PRCL, SRCL and MICH.



(Background)
We will re-position the ETMY/Y suspensions to adjust the arm lenghts during the coming vent.
To get a reasonable sensing matrix for LSC, the arm length must be adjisted within a certain precision.
So we need to know the tolerance of the arm lengths.
 

(How to estimate)
Optickle, a frequency domiain interferomtere simulator, is used to model the response of the 40m interferometer.
I buit a 40m model in Optickle, and in this model every optical distance is adjusted to the perfect length.
Then some offsets are added on the macroscopic position of ETMs to see what will happen in the LSC sensing matrix.
When putting the offsets, the amount of offsets are randomly assigned with a Gaussian distribution (see Figure.1).
Therefore the calculation is a Monte-Calro style, but this doesn't have to be a Monte-Calro
because the parameter space is only 2-dimensions (i.e. X-arm and Y-arm length) and it can be done by simply scanning the 2-dimentional parameter space.
The reason why the Monte-Calro style was chosen is because I wanted to expand this simulation
to a more general simulation which can handle PRCL, SRCL and MICH misplacements as well.
This time I ran the Monte-Calro 1000 times.
 
random2cm.png
Figure.1 History of random walk in X-Y arm lengths parameter space.
The position of ETMY and ETMX are randomly chosen with a Gaussian distribution function in every simulation.
This example was generated when \sigma_x = \sigma_y = 2 cm, where \sigma is the standard deviation of 
the Gaussian function. The number of simulation is 1000 times.
 
 
 

(Criteria)
I made two criteria for the acceptable sensing matrix as follows :
  (1) The decrease in the optical gain of the important signals (diagonal signals) must be within a factor of 3 (factor of ~ 0.5 in log scale).
  (2) MICH and SRCL signals are separated within a range of 60 - 120 deg in their demodulation phases on POP55.
 

(Results1 : sensing matrix)
Figure.2 shows the resultant sensing matrix with histograms when \sigma_x = \sigma_y = 2,
where \sigma_x, \sigma_y are the given standard deviation in the position of ETMX and ETMY.
The diagonal signals (in red-rectangular window) shows variation in their optical gain within a factor of 0.5 in log scale (factor of 3 in linear scale).
This satisfies my requirement (1) mentioned in the last section.
 
 
 
armsensMAT.png
Figure.2  A sensing matrix of the 40-m DRFPMI while changing the position of ETMX/Y by \sigma = 2 cm.
For convenience,  only REFL11, AS55, POP11 and POP55 are shown. They are the designed signal ports that
mentioned in the aLIGO LSC document (T1000298). In all the histograms, x-axis represents the optical gain in log scale in units of [W/m].
The y-axis is the number of events. The diagonal ports are surrounded by red rectangular window.
 
 
 
 
(Results2 : demodulation phase of MICH and SRCL on POP55)
Now a special attention should be payed on the MICH and SRCL signals on POP55.
Since MICH and SRCL are designed to be taken from POP55, they must be nicely separated in their demodulation phases.
Therefore the demodulation phase of MICH and SRCL has to be carefully examined.
The plot in Figure.3 is the resultant phase difference between MICH and SRCL on POP55 when \sigma_x = \sigma_y = 2 cm.
As shown in the plot the phase are always within a range of 60 - 120 deg, which satisfies my requirement (2) mentioned in the last section.
 
 
 
 
POP55phase2cm.png
 Figure.3 Difference in the demodulation phase of MICH and SRCL on POP55.
x-axis is the difference in the demodulation phase of MICH and SRCL, and y-axis the number of events.
 
 

(Notes on the Optickle model)
Optickle that I used is the one downloaded from the MIT CVS server and I believe this is the latest version.
In my current simulation I omitted some foldng mirrors including PR3, SR2 and SR3.
If those mirrors are added on the model, loss from those mirrors will affect the build up powers in all the cavities and hence changes the sensinag matrix somewhat.
I assumed that each optic has loss of 50 ppm in its HR surface.
Input power, after the MC, of 1 W is assumed.
The modulation depth are all 0.1 rad for 11MHz and 55MHz.
The model files were uploaded on the MIT CVS server and files reside under /export/cvs/iscmodeling/40m/fullIFO_Optickle.
More information about the CVS server can be found on aLIGO wiki.
 
  5081   Mon Aug 1 11:46:56 2011 ranaSummaryLSCTolerance of Arm length = 2 cm

wow. This Monte-Carlo matrix is one of the most advanced optical modeling things I have ever seen. We never had this for any of the interferometers before.

  5273   Sat Aug 20 00:42:22 2011 KeikoUpdateLSCTolerance of PRC, SRC, MICH length = 2 mm ?

 Keiko, Kiwamu

 I have run Kiwamu's length tolerance code (in CVS iscmodeling, ArmTolerance.m & analyseArmTorelance.m ) for the vertex ifo.

In his previous post, he monte-carlo-ed the arm lengths and saw the histogram of the sensing matrix and the demodulation phase between POP55 MICH and POP55 SRCL. From these plots, he roughly estimated that the tolerance is about 1 cm (sigma of the rondom gaussian) and in that case POP55 MICH and SRCL is separated by the demodulation phase 60-150 degrees.

This time I put the length displacements of random gaussian on PRC, SRC, MICH lengths at the same time (Fig.1).

 

fig3.png

Fig. 1. History of random walk in PRC, SRC, MICH lengths parameter space. Same as Kiwamu's previous post, The position of the three degrees are randomly chosen with a Gaussian distribution function in every simulaton. This example was generated when \sigma = 1 cm for all the three lengths, where \sigma is the standard deviation of  the Gaussian function. The number of simulation is 1000 times.

When the sigma is 1 cm, we found that the sensing matrix is quite bad if you look at Fig. 2. In Fig.2 row POP55, although the desired degrees of freedoms are MICH and SRCL, they have quite a bit of variety. Their separation in the demodulation phase is plotted in Fig.3. The separation in the demodulation phase varies from 40 degrees to 140 degrees, and around 270 degrees. It is not good as ideally we want it to be 90.

drawing.pdf

Fig. 2 Histgram of the sensing signal power in the matrix when 1 cm sigma rondom gaussian is applied on PRC, SRC, MICH lengths. x axis it the signal power in log10.

 

 

fig4.png

  Fig.3 POP55 MICH and POP55 SRCL separation with the displacement sigma 1 cm. 

  

 Kiwamu suspected that PRC length as more strict tolerance than other two (SRC, MICH) for POP55, as 55MHz is fast and can be sensitive to the arm length change. So I ran the same monte-carlo with SRC, MICH displacements but no PRC displacements when sigma is the same, 1cm. The results were almost same as above, nothing obvious difference.

 

With 2mm sigma, the signal power matrix and the POP55 MICH and POP55 SRCL separation in the demodulation phase look good (Fig. 4 and Fig. 5). 

fig1.pdf 

 Fig.4 Signal power matrix when PRC, SRC, MICH lengths fractuate with random gaussian distribution with 2mm sigma. The signal powers are shown in log10 in x axis, and they do not vary very much in this case.

fig4.png 

 Fig.5 POP55 MICH and POP55 SRCL separation with the displacement sigma 2 mm. The separation of the two signal is 60-90 degrees, much better than when sigma is 1 cm. We may need to check 60 degree separation is really ok or not.

 

PRC SRC MICH lengths tolerances of 2 mm in the real world will be very difficult ! 

Next I will check what happens on 3f signals.

 

 

Quote:

 Required arm length = 37.7974 +/- 0.02 [m]

This is a preliminary result of the estimation of the Arm length tolerance.

This number was obtained from a simulation based on Optickle.
Note that the simulation was done by considering misplacements in only the arm lengths while keeping PRCL, SRCL and MICH at the ideal lengths.
Therefore the tolerance will be somewhat tighter if misplacements in the central part are taken into account.

Next : check 3f signals, and include misplacements in PRCL, SRCL and MICH.

 

 



 
 
armsensMAT.png
Figure.2  A sensing matrix of the 40-m DRFPMI while changing the position of ETMX/Y by \sigma = 2 cm.
For convenience,  only REFL11, AS55, POP11 and POP55 are shown. They are the designed signal ports that
mentioned in the aLIGO LSC document (T1000298). In all the histograms, x-axis represents the optical gain in log scale in units of [W/m].
The y-axis is the number of events. The diagonal ports are surrounded by red rectangular window.
 
 
 
 
(Results2 : demodulation phase of MICH and SRCL on POP55)
Now a special attention should be payed on the MICH and SRCL signals on POP55.
Since MICH and SRCL are designed to be taken from POP55, they must be nicely separated in their demodulation phases.
Therefore the demodulation phase of MICH and SRCL has to be carefully examined.
The plot in Figure.3 is the resultant phase difference between MICH and SRCL on POP55 when \sigma_x = \sigma_y = 2 cm.
As shown in the plot the phase are always within a range of 60 - 120 deg, which satisfies my requirement (2) mentioned in the last section.
 
 
 
 
POP55phase2cm.png
 Figure.3 Difference in the demodulation phase of MICH and SRCL on POP55.
x-axis is the difference in the demodulation phase of MICH and SRCL, and y-axis the number of events.
 
 

 

 

  5292   Tue Aug 23 17:51:37 2011 KeikoUpdateLSCTolerance of PRC, SRC, MICH length = 2 mm ?

Keiko, Kiwamu

We noticed that we have used wrong code for MICH degree of freedom for both of the ELOG entries on this topic (cavity lengths tolerance search). It will be modified and posted soon.

  5334   Fri Sep 2 04:41:35 2011 KeikoUpdateLSCTolerance of PRC, SRC, MICH length = 5 mm ?

 Keiko, Kiwamu

Length tolerance of the vertex part is about 5 mm.

Sorry for my procrastinating update on this topic. In my last post, I reported that the length tolerance of the vertex ifo would be 2mm, based on Kiwamu's code on CVS. Then we noticed that the MICH degrees of freedom was wrong in the code. I modified the code and ran again. You can find the modified codes on CVS (40m folder, analyzeDRMITolerance3f.m and DRMITolerance.m)

In this code, the arm lengths were kept to be ideal while some length offsets of random gaussian distribution were added on PRCL, SRCL and MICH lengths. The iteration was 1000 times for each sigma of the random gaussian distribution. The resulting sensing matrix is shown as histogram. Also, a histogram of the demodulation phase separation between MICH and SRCL is plotted by this code, as these two length degrees of freedom will be obtained by one channel separated by the demodulation phase. We check this separation because you want to make sure that the random length offsets does not make the separation of these two signals close.

The result is a bit different from the previous post, in the better way! The length tolerance is about 5 mm for the vertex ifo. Fig.1 shows the sensing matrix. Although signal levels are changed by the random offsets, only few orders of magnitude is changed in each degrees of freedom. Fig.2 shows that the signal separation between MICH and SRCL at  POP55 varies from  55 to 120 degrees, which may be OK. If you have 1cm sigma, it varies from 50 degrees to 150 degrees.

MATRIX.png

Fig. 1 Histgram of the sensing matrix including 3f channels, when sigma is 5mm. Please note that the x-axis is in long 10.

dphase.png

 Fig. 2 Histogram of the demodulation phase difference between MICH and SRCL, when sigma is 5 mm. To obtain the two signals independently, 90 is ideal. With the random offsets, the demodulation phase difference varies from 55 degrees to 120 degrees.

My next step is to run the similar code for LLO. 

  17000   Wed Jul 13 17:30:19 2022 KojiUpdateCDSToo huge script_archive

I wanted to check the script archive to see some old settings. I found that the script archive inflated to huge volume (~1TB).
The size of the common NFS volume (/cvs/cds) is 3TB. So it is really significant.

- The scripts living in /opt/rtcds/caltech/c1/scripts are archived daily in /cvs/cds/caltech/scripts_archive as bz2 files. This is done by crontab of megatron (see https://wiki-40m.ligo.caltech.edu/Computers_and_Scripts/CRON)

- In fact, the script folder (say old script folder) /opt/rtcds/caltech/c1/scripts has the size of 10GB. And we have the compressed copy of thi s everyday.

- This large script folder is due to a couple of huge files/folders listed below

  • (scripts)/MEDMtab is 5.3GB / This is probably related to the web MEDM view (on nodus) but I don't think the web page is not updated. (i.e. the images are unused)
  • (scripts)/MC/logs/AutoLocker.log 2.9GB / This is just the accumulated MC autolocker log.
  • (scripts)/GigE 780M / This does not look like scripts but source and object files
  • (scripts)/Admin/n2Check.log 224M / This is important but increases every minute.
  • (scripts)/ZI 316MB / Zurich Instrument installation. This should not be here.

Here I propose some changes.
For the script archive

  • We can remove most of the scripts for the past (say ~2019). We leave an archive file per month.
  • For the scripts in 2020, we leave a weekly archive.
  • For 2021 and 2022, we leave all the archive files.

For the existing large files/folders

  • MEDMtab: the stored files are redundant with the burt snapshots. Remove the image files. Also, we want to move the image-saving location.
  • Autolocker.log: simply zap it
  • n2Check.log: we should move the saving location
  • GigE /ZI: they need a new home where the daily copy is not taken.
  17043   Thu Jul 28 15:11:59 2022 KojiUpdateCDSToo huge script_archive

As a result of the following work, the file volume of /cvs/cds was reduced from 3.2TB to 2.2TB, and /opt/rtcds/caltech/c1/scripts was reduced from 10GB to 1.5GB


/cvs/cds/caltech/scripts_archive was cleaned up. Now the archive files are reduced to have:

  • every month 1st day from 2005 to 2018/12
  • every ten days (1, 11, 21) for 2019 and 2020
  • everyday for 2021 and 2022

(scripts)/MEDMtab/image was deleted. I can be restore back from one of the script_archive files.

(scripts)/MC/logs/AutoLocker.log was just deleted and refreshed. For the past settings, we can refer autoburt snapshots or script_archive files.

(scripts)/Admin/n2Check.log

  • It turned out that the frequency of the check was reduced to once per 10min on Sep 9th, 2021 (unelogged activity).
  • The volume of the text since then was not much volume. So I deleted the lines before this date. And the file size is <7MB now.

(scripts)/ZI was moved to /cvs/cds/apps

/opt/rtcds/caltech/c1/burt/autoburt/snapshots

  • 2018, 2019, 2020 snapshots were archived in tar.gz.
  • These snapshots were then deleted

 

ELOG V3.1.3-