[Jenne, Diego]

Life would be easier with the UGF servos working.  As Diego already elogged, we aren't sure why the demod phases are changing, but that is certainly causing the I-signals to dip below zero, which the log function can't handle (there is a limiter before the log, so that the signal can't go below 1e-3).  Anyhow, this is causing the UGF servos to freak out, so we have not been using them for tonight's locking.

Our goal tonight was to see if we could introduce a nice big MICH offset, and then lower the CARM offset while keeping the arms locked on ALS.  We hope to see some kind of sign of a PDH signal in some RF PD. 

In the end, the highest we got to was -460 MICH offset counts, which we think is about 29nm (if our rough calibration is accurate). The MICH half fringe should be 188nm. With this offset, we got down to 0.3 CARM offset counts while locked on ALS.  We think that this is around 300pm, plus or minus a lot.  Note that while yesterday I had a pretty easy time getting to -660 counts of MICH offset, tonight I struggled to get past -200.  The only way we ended up getting farther was by lowering the CARM offset.  Although, as I type this, I realized that last night's work already had a lower CARM offset, so maybe that's key to being able to increase the MICH offset. 

We watched REFL11I and REFL11I/(TRX+TRY) on striptool, but we didn't see any evidence of a PDH signal.  We lost lock when I tried to transition CARM over to REFLDC, but I wasn't careful about my offset-setting, so I am not convinced that REFLDC is hopeless.

So.  Tonight, we didn't make any major locking progress (the MC started being fussy for about an hour, right after I ran the LSC offsets script, just before we started locking in earnest).  However, we have some ideas from talking with Rana about directions to go:

* Can we transition CARM over to REFL11I, and then engage the AO path?

* Then, while the MICH offset is still large, can we transition DARM over to POX or POY, actuating on a single arm?  If CARM is totally suppressed, this is DARM-y.  If CARM doesn't have the AO path yet, this is halfsy-halfsy, but maybe we don't care.

* Then, can we lower the MICH offset and transition back to a REFLQ signal?

* Separately, it seemed like we kept losing PRC lock due to PRC motion.  If the MICH offset is very large, are we sideband-limited at the POP port, such that we can use the POP DC QPD?  Is it even worth it?

MICH calibration:

A single mirror (ITM) moving by lambda/2, in the MICH-only situation is the full range, from bright to dark fringe.  So, half fringe should be lambda/4, or about 133nm.  If we are thinking about pushing on the BS, there's an extra factor of sqrt(2), so I think the half fringe should be at 188nm of BS motion.

When we had MICH locked on ASDC/POPDC, we put in a line at 143.125Hz, at 20 counts to (0.5*BS-0.2625*PRM), so a total of 10 counts to the BS at 143Hz.  Given the BS calibration in http://nodus.ligo.caltech.edu:8080/40m/8242, this is 10.1pm of actuation.  We saw a line in the error signal of 0.1 counts, so we infer that the MICH error signal of ASDC/POPDC has a calibration of 94pm/count. This number was invariant over a few different MICH offsets, although the ones I measured at were all below about 100 counts of MICH offset, so maybe this number changes as we start to get farther from the MICH dark fringe.

IFO left flashing (all mirrors aligned except SRM) in case anyone wants fresh data for that.

MC1     28.21625 kHz
MC2     28.036   kHz
MC3     28.21637 kHz

For testing the new IR ALS noise, we had decided that we would like to use the differential output of the demodulated ALS beat signal, as opposed to a single-ended output, as measurements suggested the former to be a lower noise configuration than the latter. For this purpose, Koji and I acquired a couple of old AA boards from the WB electronics shop. These are however, rev2 of the board, whereas the latest version is v6. The main difference between v2 and v6 is that (i) the THS4131 instrumentation amplifier has the Vocm pin grounded in v6 but is floating in v2 and (ii) the buffer opamps are AD8622 in v6 but are AD8672 in v2. But in fact, the boards we have are stuffed with AD8682. 

I talked to Rich on Friday, and he seemed to think the AD8672 didn't have any issues noise-wise, the main reason they changed it was because its power consumption was high, and was causing overheating when several of these 1U chassis were packed closely together in an electronics rack. But the AD8682, which is what we have, has comparable power consumption to the AD8622. It is however a JFET opamp, and the voltage noise is a bit higher than the AD8622. 

I am sure there is a way to LISO model a differential output opamp like the THS4131, but I thought I'd simulate the noise in LTSPICE instead. But I couldn't get that to work. So instead, I just measured the transfer function and noise of a single channel, for which Koji had expertly hacked together a custom shorting of the THS4131 Vocm pin to ground. Attachments #1 and #2 show the measurement. All looks good. Note that the phase is 180 at DC because I had hooked up the input signal opposite to what it should have been. The voltage noise of the differential outputs (each measured w.r.t. ground, with both inputs shorted to ground by a short patch cable) at 10 Hz is <100nV/rtHz, and the ADC noise is expected to be ~1uV/rtHz, so I think this is fine.

Conclusion: I think for the ALS test, we can just use the AA board in this config without worrying too much about replacing the buffer stage opamps, even though we've ordered 100pcs of AD8622.

Addendum 7 Mar 2018 11am: As per this document, the output noise of the AA board should be <75nV/rtHz from 10 Hz-50 kHz. So maybe the AD8682 noise is a little high after all. I've gotten the LTSpice model working now, will post the comparison of modelled output noise for various combinations here shortly.

Here are the plots. Comments:

1. Measurement and model agree quite well .
2. Of the 3 OpAmps, the ones installed seem to be the noisiest (per model). 
3. Despite #2, I don't think it is critical to replace the buffer opamps as we only win by ~10nV/rtHz in the 300-10kHz range.
4. I don't understand the spec given in T070146. It says the noise everywhere between 10Hz-50kHz should be <75nV/rtHz. But even the model suggests that at 10Hz, the noise is ~250nV/rtHz for any choice of buffer opamp, so that's a factor of 3 difference which seems large. Maybe I made a mistake in the model but the agreement between measurement and model for the AD8682 choice gives me confidence in the simulation. LTSpice files used are in Attachment #3. Could also be an artefact of the way I made the measurement - between an output and ground instead of differentially...

I like LTspice for such modeling - the GUI is nice to have (though I personally think that typing out a nodal file a la LISO is faster), and compared to LISO, I think that the LTspice infrastructure is a bit more versatile in terms of effects that can be modeled. We can also easily download SPICE models for OpAmps from manufacturers and simply add them to the library, rather than manually type out parameters in opamp.lib for LISO. But the version available for Mac is somewhat pared down in terms of the UI, so I had to struggle a bit to find the correct syntax for the various simulation commands. The format of the exported data is also not as amenable to python plotting as LISO output files, but i'm nitpicking...

Summary:

I injected some sensing lines and measured their responses in the various photodiodes, with the interferometer in a few different configurations. The results are summarized in Attachments #1 - #3. Even with the PRMI (no arm cavities) locked on 1f error signals, the MICH and PRCL signals show up in nearly the same quadrature in the REFL port photodiodes, except REFL165. I am now thinking if the output (actuation) matrix has something to do with this - part of the MICH control signal is fed back to the PRM in order to minimize the appearance of the MICH dither in the PRCL error signal, but maybe this matrix element is somehow horribly mistuned?

Details:

Attachment #1:

• ETMs were misaligned and the PRMI was locked with the carrier resonant in the cavity (i.e. sidebands reflected).
• The locking scheme was AS55_Q --> MICH and REFL11_I --> PRCL.

Attachment #2:

• The PRFPMI was locked. The vertex DoFs were still under control using 3f error signals (REFL165_I for PRCL and REFL165_Q for MICH).
• Still, the MICH/PRCL degeneracy in all photodiodes except REFL165 persists.

Attachment #3:

• Nearly identical configuration to Attachment #2.
• The main difference here is that I applied some offsets to the MICH and PRCL error points.
• The offsets were chosen so that the appearance of a ~300 Hz dither in the length of MICH/PRCL was nulled in the AS110_Q / POP22_I signals respectively.
• For the latter, the appearance of this peak in the POP110_I signal was also nulled, as it should be if our macroscopic PRC length is set correctly.
• The offsets that best nulled the peak were 110 cts for PRCL, 25 cts for MICH. The measured sensing response is 1e12 cts/m for PRCL in REFL165_I and 9.2e11 cts/m for MICH in REFL165_Q. So these offsets, in physical units, are: 110 pm for PRCL and 27 pm for MICH. They seem like reasonable numbers to me - the PRC linewidth is ~7.5 nm, so the detuning without any digital offset applied is only 1.5% of the linewidth.
• Note that I changed the POP22/POP110 demod phases to maximize the signal in the I quadrature. The final numbers were -124 degrees / -10 degrees respectively.
• Yet another piece of evidence suggesting these were the correct offsets is that the DC value of POX and POY were zero on average after these offsets were applied.
• However, the MICH/PRCL responses in the 1f REFL port photodiodes remain nearly degenerate.

Some other mysteries that I will investigate further:

1. While POP22 indicated stable buildup of 11 MHz power in the PRC, I couldn't make any sense of the AS110 signals at the dark port - there was large variation of the signal content in the two quadratures, so unlike the POP signals, I couldn't find a digital demod phase that consistently had all the signal in one of the two quadratures. This is all due to angular fluctuations?
2. My ASC simulations suggest that the POP QPD is a poor sensor of PRM motion when the PRFPMI is locked. However, I find that turning on a feedback loop with the POP QPD as a sensor and the PRM as the actuator dramatically reduces the low-frequency fluctuations of the arm cavity carrier buildup. 🤔

I blew the long lock last night because I forgot to not clear the ASS offsets when trying to find the right settings for running the ASS system at high power. Will try again tonight...

Our existing 300 series SS plungers from McMastercar #8476A43 are silver plated as Atm2 shows.

Problems:  1, they become magnetized after years being close to the magnets

                     2, they oxidize by time so it is hard to turn them

I looked around to replace them.

Titanium body, nose and beryllium copper spring. None magnetic for UHV enviorment.

Can be made in 7 weeks at an UNREASONABLE $169.00 ea at quantity of 50

We have thousands of miles of BNC cables in the lab but we still don't find one when we need it. I decided to solve the problem.

This morning I tried to tidy up the several cable rack the we have in the lab.

i tried to dedicate each rack to a speecific type of cables: special cables, hand made cables, BNCs, LEMOs, etc.

In particular I tryed to concentrate BNC cable of several lengths on the rack near by the ITMX chamber.

People are invited to preserve the organization.

This would be a good trial once you put the label "BNC only" on the wall.

Done! Check it out.

I measured the transfer functions in the delay line cables, and then calculated the time delay from that.

The first cable had a time delay of 1272 ns and the second had a time delay of 1264 ns. 

Below are the plots I created to calculate this. There does seem to be a pattern in the residual plots however, which was not expected. 

The R-Square parameter was very close to 1 for both fits, indicating that the fit was good. 

 A time delay can be modeled as the exponential transfer function :  e^(-sT_d)  as seen HERE . Therefore the slope of the phase gives us the time delay.

A RG405 coaxial cable, exactly 5.5 meters in length, was fit to an ideal delay function e^(-sT_d) , with T_d = 150 ns.

The plots shows the actual data, fit data and data after correction using the ideal model stated above.

Conclusion:

Delay in RG405 cables is approximately 27.27 ns per meter. This value can be used to correct the phase in measurements of transimpedance for each PD by dividing out the ideal transfer function for time delay.

[EDIT: This looks like we have about 12 % the speed of light inside the RF cables. Too small to be true. I will check tomorrow if the Network analyzer itself has some delay and update this value.]

The varying attenuation of about 1dB due to the cable is not compensated by this. We need to separately include this.

Things to do:

1) Get the length of RF cables that is being used by each PD, so that the compensation can be made.

2) Calculate the attenuation and delay caused by RF multiplexer and Demodulator boards. Include these in the correction factor for transimpedance measurements. 

A time delay can be modeled as the exponential transfer function :  e^(-sT_d)  as seen HERE . Therefore the slope of the phase gives us the time delay.

The transfer function of RF multiplexer in rack 1Y1 (NI PXI-2547) was fit to an ideal delay function e^(-sT_d) , with T_d = 59 ns.

The plots shows the actual data, fit data and data after correction using the ideal model stated above.

Conclusion:

Delay the RF Multiplexer is approximately 59 ns. This value can be used to correct the phase in measurements of transimpedance for each PD by dividing out the ideal transfer function for time delay.

I tried to read a bit and understand the NTP synchronization implementation in FE computers. I'm quite sure that NTP synchronization should be 'yes' if timesyncd are running correctly in the output of timedatectl in these computers. As Koji reported in 15791, this is not the case. I logged into c1lsc, c1sus and c1ioo and saw that RTC has drifted from the software clocks too which does not happen if NTP synchronization was active. This would mean that almost certainly, if the computers are rebooted, the synchronization will be lost and the models will fail to come online.

My current findings are the following (this should be documented in wiki once we setup everything):

• nodus is running a NTP server using chronyd. One can check the configuration of this NTP serer in /etc/chornyd.conf
• fb1 is running an NTP server using ntpd that follows nodus and an IP address 131.215.239.14. This can be seen in /etc/ntp.conf.
• There are no comments to describe what this other server (131.215.239.14) is. Does the GC network have an NTP server too?
• c1lsc, c1sus and c1ioo all have systemd-timesyncd.service running with configuration file in /etc/systemd/timesyncd.conf.
• The configuration file set Servers=ntpserver but echo $ntpserver produces nothing (blank) on these computers and I've been unable to find anyplace where ntpserver is defined.
• In chiara (our name server), the name server file /etc/hosts does not have any entry for ntpserver either.
• I think the problem might be that these computers are unable to find the ntpserver as it is not defined anywhere.

The solution to this issue could be as simple as just defining ntpserver in the name server list. But I'm not sure if my understanding of this issue is correct. Comments/suggestions are welcome for future steps.

131.215.239.14 looks like Caltech's NTP server (ntp-02.caltech.edu)
https://webmagellan.com/explore/caltech.edu/28415b58-837f-4b46-a134-54f4b81bee53

I can't say it is correct or not as I did not make the survey at your level. I think you need a few tests of reconfiguring and restarting the NTP clients to see if time synchronization starts. Because the local time is not regulated right now anyway, this operation is safe I think.

I added ntpserver as a known host name for address 192.168.113.201 (fb1's address where ntp server is running) in the martian host list in the following files in Chiara:

/var/lib/bind/martian.hosts
/var/lib/bind/rev.113.168.192.in-addr.arpa

Note: a host name called ntp was already defined at 192.168.113.11 but I don't know what computer this is.

Then, I restarted the DNS on chiara by doing:

sudo service bind9 restart

Then I logged into c1lsc and c1ioo and ran following:

controls@c1ioo:~ 0$ sudo systemctl restart systemd-timesyncd.service

controls@c1ioo:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
   Active: active (running) since Fri 2021-08-20 07:24:03 UTC; 53s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 23965 (systemd-timesyn)
   Status: "Idle."
   CGroup: /system.slice/systemd-timesyncd.service
           └─23965 /lib/systemd/systemd-timesyncd

Aug 20 07:24:03 c1ioo systemd[1]: Starting Network Time Synchronization...
Aug 20 07:24:03 c1ioo systemd[1]: Started Network Time Synchronization.
Aug 20 07:24:03 c1ioo systemd-timesyncd[23965]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:24:35 c1ioo systemd-timesyncd[23965]: Using NTP server 192.168.113.201:123 (ntpserver).
controls@c1ioo:~ 0$ timedatectl
      Local time: Fri 2021-08-20 07:25:28 UTC
  Universal time: Fri 2021-08-20 07:25:28 UTC
        RTC time: Fri 2021-08-20 07:25:31
       Time zone: Etc/UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

The same output is shown in c1lsc too. The NTP synchronized flag in output of timedatectl command did not change to yes and the RTC is still 3 seconds ahead of the local clock.

Then I went to c1sus to see what was the status output before rstarting the timesyncd service. I got folloing output:

controls@c1sus:~ 0$ sudo systemctl status systemd-timesyncd.service -l
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
   Active: active (running) since Tue 2021-08-17 04:38:03 UTC; 3 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 243 (systemd-timesyn)
   Status: "Idle."
   CGroup: /system.slice/systemd-timesyncd.service
           └─243 /lib/systemd/systemd-timesyncd

Aug 20 02:02:18 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 02:36:27 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:10:35 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 03:44:43 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:18:51 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 04:53:00 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 05:27:08 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:01:16 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 06:35:24 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 20 07:09:33 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).

This actually shows that the service was able to find ntpserver correctly at 192.168.113.201 even before I changed the name server file in chiara. So I'm retracting the changes made to name server. They are probably not required.

The configuration files for timesynd.conf are read only even with sudo. I tried changing permissions but that did not work either. Maybe these files are not correctly configured. The man page of timesyncd  says to use field 'NTP' to give the ntp servers. Our files are using field 'Servers'. But since we are not getting any error message, I don't think this is the issue here.

I'll look more into this problem.

I read on some stack exchange that 'NTP synchornized' indicator turns 'yes' in the output of command timedatectl only when RTC clock has been adjusted at some point. I also read that timesyncd does not do the change if the time difference is too much, roughly more than 3 seconds.

So I logged into all FE machines and ran sudo hwclock -w to synchronize them all to the system clocks and then waited if the timesyncd does any correction on RTC. It did not. A few hours later, I found the RTC clocks drifitng again from the system clocks. So even if the timesynd service is running as it should, it si not performing time correction for whatever reason.

Maybe we should try to use some other service?

Related elog thread: 16286

I didn't really achieve anything but I'm listing what I've tried.

• I know now that the timesyncd isn't working because systemd-timesyncd is known to have issues when running on a read-only file system. In particular, the service does not have privileges to change the clock or drift settings at /run/systemd/clock or /etc/adjtime.
• The workarounds to these problems are poorly rated/reviews in stack exchange and require me to change the /etc/systmd/timesyncd.conf file but I'm unable to edit this file.
• I know that Paco was able to change these files earlier as the files are now changed and configured to follow a debian ntp pool server which won't work as the FEs do not have internet access. So the conf file needs to be restored to using ntpserver as the ntp server.
• From system messages, the ntpserver is recognized by the service as shown in the second part of 16285. I really think the issue is in file permissions. the file /etc/adjtime has never been updated since 2017.
• I got help from Paco on how to edit files for FE machines. The FE machines directories are exported from fb1:/diskless/root.jessie/
• I restored the /etc/systmd/timesyncd.conf file to how it as before with just servers=ntpserver line. Restarted timesyncd service on all FEs,I tried a few su the synchronization did not happen.
• I tried a few suggestions from stackexchange but none of them worked. The only rated solution creates a tmpfs directory outside of read-only filesystem and uses that to run timesyncd. So, in my opinion, timesyncd  would never work in our diskless read-only file system FE machines.
• One issue in an archlinux discussion ended by the questioner resorting to use opennptd from openBSD distribution. The user claimed that opennptd is simple enough that it can run ntp synchornization on a read-only file system.
• Somehwat painfully, I 'kind of' installed the openntpd tool in the fb1:/diskless/root.jessie directory following directions from here. I had to manually add user group and group for the FEs (which I might not have done correctly). I was not able to get the openntpd daemon to start properly after soe tries.
• I restored everything back to how it was and restarted timesyncd in c1sus even though it would not do anything really.

tl;dr: NTP servers and clients were never synchronized, are not synchronizing even with ntp... nodus is synchronized but uses chronyd; should we use chronyd everywhere?

Spent some time investigating the ntp synchronization. In the morning, after Anchal set up all the ntp servers / FE clients I tried restarting the rts IOPs with no success. Later, with Tega we tried the usual manual matching of the date between c1iscex and fb1 machines but we iterated over different n-second offsets from -10 to +10, also without success.

This afternoon, I tried debugging the FE and fb1 timing differences. For this I inspected the ntp configuration file under /etc/ntp.conf in both the fb1 and /diskless/root.jessie/etc/ntp.conf (for the FE machines) and tried different combinations with and without nodus, with and without restrict lines, all while looking at the output of sudo journalctl -f on c1iscey. Everytime I changed the ntp config file, I restarted the service using sudo systemctl restart ntp.service . Looking through some online forums, people suggested basic pinging to see if the ntp servers were up (and broadcasting their times over the local network) but this failed to run (read-only filesystem) so I went into fb1, and ran sudo chroot /diskless/root.jessie/ /bin/bash to allow me to change file permissions. The test was first done with /bin/ping which couldn't even open a socket (root access needed) by running chmod 4755 /bin/ping then ssh-ing into c1iscey and pinging the fb1 machine successfully. After this, I ran chmod 4755 /usr/sbin/ntpd so that the ntp daemon would have no problem in reaching the server in case this was blocking the synchronization. I exited the chroot shell and the ntp daemon in c1iscey; but the ntpstat still showed unsynchronised status. I also learned that when running an ntp query with ntpq -p if a client has succeeded in synchronizing its time to the server time, an asterisk should be appended at the end. This was not the case in any FE machine... and looking at fb1, this was also not true. Although the fb1 peers are correctly listed as nodus, the caltech ntp server, and a broadcast (.BCST.) server from local time (meant to serve the FE machines), none appears to have synchronized... Going one level further, in nodus I checked the time synchronization servers by running chronyc sources the output shows

controls@nodus|~> chronyc sources
210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* testntp1.superonline.net      1  10   377   280  +1511us[+1403us] +/-   92ms
^+ 38.229.59.9                   2  10   377   206  +8219us[+8219us] +/-  117ms
^+ tms04.deltatelesystems.ru     2  10   377   23m    -17ms[  -17ms] +/-  183ms
^+ ntp.gnc.am                    3  10   377   914  -8294us[-8401us] +/-  168ms

I then ran chronyc clients to find if fb1 was listed (as I would have expected) but the output shows this --

Hostname                   Client    Peer CmdAuth CmdNorm  CmdBad  LstN  LstC
=========================  ======  ======  ======  ======  ======  ====  ====
501 Not authorised

So clearly chronyd succeeded in synchronizing nodus' time to whatever server it was pointed at but downstream from there, neither the fb1 or any FE machines seem to be synchronizing properly. It may be as simple as figuring out the correct ntp configuration file, or switching to chronyd for all machines (for the sake of homogeneity?)

I attempted to install chrony and run it on one of the FE machines. It didn't work and in doing so, I lost the working NTP client service on the FE computers as well. Following are some details:

• I added the following two mirrors in the apt source list of root.jessie at /etc/apt/sources.list
  deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
  deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free
• Then I installed chrony in the root.jessie using
  sudo apt-get install chrony
  • I was getting an error E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory) . To fix this, I had to run:
    sudo mount -t devpts none "$rootpath/dev/pts" -o ptmxmode=0666,newinstance
    sudo ln -fs "pts/ptmx" "$rootpath/dev/ptmx"
  • Then, I had another error to resolve.
    Failed to read /proc/cmdline. Ignoring: No such file or directory
    start-stop-daemon: nothing in /proc - not mounted?
    To fix this, I had to exit to fb1 and run:
    sudo mount --bind /proc /diskless/root.jessie/proc
  • With these steps, chrony was finally installed, but I immediately saw an error message saying:
    Starting /usr/sbin/chronyd...
    Could not open NTP sockets
• I figured this must be due to ntp running in the FE machines.  I logged into c1iscex and stopped and disabled the ntp service:
  sudo systemctl stop ntp
  sudo systemctl disable ntp
  • I saw some error messages from the above coomand as FEs are read only file systems:
    Synchronizing state for ntp.service with sysvinit using update-rc.d...
    Executing /usr/sbin/update-rc.d ntp defaults
    insserv: fopen(.depend.stop): Read-only file system
    Executing /usr/sbin/update-rc.d ntp disable
    update-rc.d: error: Read-only file system
  • So I went back to chroot in fb1 and ran the two command sabove that failed:
    /usr/sbin/update-rc.d ntp defaults
    /usr/sbin/update-rc.d ntp disable
  • The last line gave the output:
    insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
    insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
  • I igored this and moved forward.
• I copied the chronyd.service from nodus to the chroot in fb1 and configured it to use nodus as the server. The I started the chronyd.service

  sudo systemctl status chronyd.service
  but got the saem issue of NTP sockets.

  â—Â� chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
     Active: failed (Result: exit-code) since Tue 2021-08-24 21:52:30 PDT; 5s ago
    Process: 790 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=1/FAILURE)

  Aug 24 21:52:29 c1iscex systemd[1]: Starting NTP client/server...
  Aug 24 21:52:30 c1iscex chronyd[790]: Could not open NTP sockets
  Aug 24 21:52:30 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
  Aug 24 21:52:30 c1iscex systemd[1]: Failed to start NTP client/server.
  Aug 24 21:52:30 c1iscex systemd[1]: Unit chronyd.service entered failed state.

• I tried a few things to resolve this, but couldn't get it to work. So I gave up on using chrony and decided to go back to ntp service atleast.

• I stopped, disabled and checked status of chrony:
  sudo systemctl stop chronyd
  sudo systemctl disable chronyd
  sudo systemctl status chronyd
  This gave the output:

  â—Â� chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled)
     Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:07 PDT; 25s ago

  Aug 24 22:09:07 c1iscex systemd[1]: Starting NTP client/server...
  Aug 24 22:09:07 c1iscex chronyd[2490]: Could not open NTP sockets
  Aug 24 22:09:07 c1iscex systemd[1]: chronyd.service: control process exited, code=exited status=1
  Aug 24 22:09:07 c1iscex systemd[1]: Failed to start NTP client/server.
  Aug 24 22:09:07 c1iscex systemd[1]: Unit chronyd.service entered failed state.
  Aug 24 22:09:15 c1iscex systemd[1]: Stopped NTP client/server.

• I went back to fb1 chroot and removed chrony package and deleted the configuration files and systemd service files:
  sudo apt-get remove chrony

• But when I started ntp daemon service back in c1iscex, it gave error:
  sudo systemctl restart ntp
  Job for ntp.service failed. See 'systemctl status ntp.service' and 'journalctl -xn' for details.

• Status shows:

  sudo systemctl status ntp
  â—Â� ntp.service - LSB: Start NTP daemon
     Loaded: loaded (/etc/init.d/ntp)
     Active: failed (Result: exit-code) since Tue 2021-08-24 22:09:56 PDT; 9s ago
    Process: 2597 ExecStart=/etc/init.d/ntp start (code=exited, status=5)

  Aug 24 22:09:55 c1iscex systemd[1]: Starting LSB: Start NTP daemon...
  Aug 24 22:09:56 c1iscex systemd[1]: ntp.service: control process exited, code=exited status=5
  Aug 24 22:09:56 c1iscex systemd[1]: Failed to start LSB: Start NTP daemon.
  Aug 24 22:09:56 c1iscex systemd[1]: Unit ntp.service entered failed state.

• I tried to enable back the ntp service by sudo systemctl enable ntp. I got similar error messages of read only filesystem as earlier.
  Synchronizing state for ntp.service with sysvinit using update-rc.d...
  Executing /usr/sbin/update-rc.d ntp defaults
  insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
  insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
  insserv: fopen(.depend.stop): Read-only file system
  Executing /usr/sbin/update-rc.d ntp enable
  update-rc.d: error: Read-only file system

  • I went back to chroot in fb1 and ran:
    /usr/sbin/update-rc.d ntp defaults
    insserv: warning: current start runlevel(s) (empty) of script `ntp' overrides LSB defaults (2 3 4 5).
    insserv: warning: current stop runlevel(s) (2 3 4 5) of script `ntp' overrides LSB defaults (empty).
    and
    /usr/sbin/update-rc.d ntp enable

• I came back to c1iscex and tried restarting the ntp service but got same error messages as above with exit code 5.

• I checked c1sus, the ntp was running there. I tested the configuration by restarting the ntp service, and then it failed with same error message. So the remaining three FEs, c1lsc, c1ioo and c1iscey have running ntp service, but they won't be able to restart.

• As a last try, I rebooted c1iscex to see if ntp comes back online nicely, but it doesn't.

Bottom line, I went to try chrony in the FEs, and I ended up breaking the ntp client services on the computers as well. We have no NTP synchronization in any of the FEs.

Even though Paco and I are learning about the ntp and cds stuff, I think it's time we get help from someone with real experience. The lab is not in a good state for far too long.

Jamie told me to use chroot to log in into the chroot jail of debian os that are exported for the FEs and install ntp there. I took following steps at the end of which, all FEs have NTP synchronized now.

• I logged into fb1 through nodus.
• chroot /diskless/root.jessie /bin/bash took me to the bash terminal for debian os that is exported to all FEs.
• Here, I ran sudo apt-get install ntp which ran without any errors.
• I then edited the file in /etc/ntp.conf , i removed the default servers and added following lines for servers (fb1 and nodus ip addresses):
  server 192.113.168.201
  server 192.113.168.201
• I logged into each FE machine and ran following commands:
  sudo systemctl stop systemd-timesyncd.service; sudo systemctl status systemd-timesyncd.service;
  timedatectl; sleep 2;sudo systemctl daemon-reload;  sudo systemctl start ntp; sleep 2; sudo systemctl status ntp; timedatectl
  sudo hwclock -s
  • The first line ensures that systemd-timesyncd.service is not running anymore. I did not uninstall timesyncd and left its configuration file as it is.
  • The second line first shows the times of local and RTC clocks. Then reloads the daemon services to get ntp registered. Then starts ntp.service and shows it's status. Finally, the timedatectl command shows the synchronized clocks and that NTP synchronization has occured.
  • The last line sets the local clock same as RTC clock. Even though this wasn't required as I saw that the clocks were already same to seconds, I just wanted a point where all the local clocks are synchronized to the ntp server.
• Hopefully, this would resolve our issue of restarting the models anytime some glitch happens or when we need ot update something in one of them.

Edit Tue Aug 24 10:19:11 2021:

I also disabled timesyncd on all FEs using sudo systemctl disable systemd-timesyncd.service

I've added this wiki page for summarizing the NTP synchronization knowledge.

Crontab: The bug of data only plotting until 5PM is being investigated.  The crontab's final call to the summary page generator was at 5PM.  This means that the data plots were not being generated after 5PM, so clearly they never contained data from after 5PM.  In fact, the original crontab reads:

0 11,5,11,17 * * * /users/public_html/40m-summary/bin/c1_summary_page.sh 2>&1

I'm not exactly sure what inspired these entries.  The 11,5,11,17 entries are supposed to be the hours at which the program is run.  Why is it run twice at 11?  I assume it was just a typo or something.

The final call time was changed to 11:59PM in an attempt to plot the entire day's data, but this method didn't appear to work because the program would still be running past midnight, which was apparently inhibiting its functionality (most likely, the day change was affecting how the data is fetched).  The best solution is probably to just wait until the next day, then call the summary page generator on the previous day's data.  This will be implemented soon.

Calendars: Although the calendar tabs on the left side of the page were fixed, the calendars displayed at: https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/ appear to still have squished together text.  The calendar is being fetched from https://nodus.ligo.caltech.edu:30889/40m-summary/calendar/calendar.html and displayed in the page.  This error is peculiar because the URL from which the calendar is being fetched does NOT have squished together text, but the resulting calendar at 40m-summary/calendar/ will not display spaces between the text.  This issue is still being investigated.

Main Problem:

The frequency counter (FC) takes in an analog RF input(signal) and outputs the frequency of the signal(Ranging from 1 MHz- 6000 MHz) in the digital domain (into a processor). The FC samples the data with a given sample rate( user defined) which ranges from 0.1 s to 1 s(faced problems in fixing this initially).  For data acquisition, we have been using a Raspberry Pi(as a processor) which is connected to the martian network and can communicate with the computers inside the 40m.  The ultimate challenge which I faced( and been knocking my head off from past two-three weeks) is the synchronization of clocks between the Raspberry Pi and the FC i.e the clock which the FC uses to sample and dump data( every 'x' s) and the clock inside the raspberry pi( used  in the loop to wait for a particular amount of time the frequency counter takes to dump successive data).

Steps Taken:

• To address this problem, first I added an external clock circuit which monitors the Raspberry Pi and the FC to dump and read data at a particular rate(which is equal to the sampling rate of the FC)In detail: http://nodus.ligo.caltech.edu:8080/40m/10129. 
• While doing so, at first the level trigger algorithm was used which means that the external clock frequency was half as that of  the reciprocal of the sampling rate and a trigger was seen every time the level shifts from +DC to -DC(of the external square wave).
• But this did not completely mitigate the issues and there were still few issues on how quickly the ADC reads the signal and R Pi processes it.
• To minimize these issues completely, an edge trigger algorithm which detects a pos edge(rising)  of the clock was used. The clock  frequency is now equal to the reciprocal of the sampling rate. This algorithm showed better results and greatly minimized the drift of the sampling time.

Psuedo Code(code attached):

open device : FC via USB-HID;

open device : ADC via I2C;

always(for t= recording time):

            read data from ADC(external clock);

            if pos edge detected:

                    read data from FC and store it in a register;

             else read data from ADC;

end

write data stored in the register to a file( can be an Epics channel or a text file);

Results:

The attached are the plots showing the time between samples for a large number of samples taken for different sampling times of the FC. The percentage error is the percentage of standard error in the timing between two samples for the data for the entire measurement. It can be inferred that this error has been cut down to the order of ms.

To do next:

• I have started taking phase measurements( analysis and plots will follow this elog) and also the PSD plots with the improved timing characteristics.

Looks good. Now you have the internal timer to verify the external clock.
If you can realize the constant rate sampling without employing the external clock, that's quite handy.

The attached are the PSD plots with improved FC timing(with the same code as in http://nodus.ligo.caltech.edu:8080/40m/10151). More plots(Phase and PSD) to follow.

At the request of people down at LLO I've been trying to work on the reliability and speed of the GigE camera code.  In my testing, after several hours, the code would tend to lock up on the camera end.  It was also reported at LLO after several minutes the camera display would slow down, but I haven't been able to replicate that problem.

I've recently added some additional error checking and have updated to a more recent SDK which seems to help.  Attached are two plots of the frames per second of the code.  In this case, the frames per second  are measured as the time between calls to the C camera code for a new frame for gstreamer to encode and transmit.  The data points in the first graph are actually the averaged time for sets of 1000 frames.  The camera was sending 640x480 pixel frames, with an exposure time of 0.01 seconds.  Since the FPS was mostly between 45 and 55, it is taking the code roughly 0.01 second to process, encode, and transmit a frame.

During the test, the memory usage by the server code was roughly 1% (or 40 megabytes out of 4 gigabytes) and 50% of a CPU (out a total of  CPUs).

See Attachment #1. J8 was connected to a "LASTI timing slave" sitting in the rack that Chiara lives in - we don't use this for anything and I confirmed that there was no effect on the RTCDS when I pulled that fiber out. The LASTI timing slave also had a blinky that was blinking when the fiber was plugged in - which I take to believe that the slot works. 

Can we get away with just using these two available slots, J8 and J13? Do we really need three new expansion chassis?

I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.

That's great. I wonder if we can also get away with not adding new Dolphin infrastructure. I'd really like to avoid changing any IPC drivers.

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

That's great news we won't have to worry about a new timing fanout for the two new machines, c1bhd and c1sus2. And there's no plan to change Dolphin IPC drivers. The plan is only to install the same (older) version of the driver on the two new machines and plug into free slots in the existing switch.

Problem:

We had timing problems across the front ends.

Solution:

Noticing that the 1PPS reference was not blinking on the Master Timing Distribution box.  It was supposed to be getting a signal from the c0dcu1 VME crate computer, but this was not happening.

We disconnected the timing signal going into c0dcu1, coming from c0daqctrl, and connected the 1PPS directly from c0daqctrl to the Ref In for the Master Timing distribution box (blue box with lots of fibers coming out of it in 1X5).

We now have agreement in timing between front ends.

After several reboots we now have working RFM again, along with computers who agree on the current GPS time along with the frame builder.

Status:

RFM is back and testpoints should be happy.

We still don't have a working binary output for the X end.  I may need to get a replacement backplane with more than 4 slots if the 1st slot of this board has the same problem as the large boards.

I have burt restored the c1ioo, c1mcs, c1rms, c1sus, and c1scx processes, and optics look to be damped.

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

• Now, old frame data is no longer being overwritten
• For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

• The usual FE/FB power cycling did not fix the problem.
• The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
• This was off by 2 years.
• The way this worked up till now was by adding a fixed offset to this time.
  • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
  • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
  • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

1. First, we tried changing the offset in the daqdrc.fw file only.
   • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
   • This did not fix the problem.
2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
   • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
   • Added the following lines:
     /* 2018 had no leap seconds or leap days, so adjust for that */
            pHardware->gpsOffset += 31536000;
   • re-built and installed the modified symmetricom driver.
   • Checked that cat /proc/gps now yields the correct time.
   • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
   • With these steps, the frames were being written to /frames with the correct timestamp.
3. Next, we checked the timing on the FEs
   • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
   • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
   • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
   • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
   • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Summary:

The gps time mismatch between /proc/gps and gpstime seems to be resolved. However, the 0x4000 DC errors still persist. It is not clear to me why.

Details:

On the phone with J Hanks on Friday, he reminded me that c1sus seems to be the only machine with an IRIG-B timing card installed. I can't find the elog but I remembered that Jamie, ericq and I had done this work in 2016 (?), and I also remembered Jamie saying it wasn't working exactly as expected. Since the DAQ was working fine before this card was installed, and since there are no problems with the recording of channels from the other four FE machines without this card installed, I decided to simply pull out the card from the expansion chassis. The card has been stored in the CDS/FE cabinet along the Y arm for now. There was also a cable that interfaces to the card which brings over the 1pps from the GPS unit, which has also been stored in the CDS/FE cabinet.

This seems to have resolved the mismatch between the gpstime reported by cat /proc/gps and the gpstime commands - Attachment #1 (the <1 second mismatch is presumably due to the deadtime between commands). However, the 0x4000 DC errors still persist. I'll try the full power cycle of FEs and FB which has fixed this kind of error in the past, but apart from that, I'm out of ideas.

Update 1215:

Following the instructions in this elog did not fix the problem. The problem seems to be with the daqd_fw service, which reports the following:

Update 1530:

The frame-writer error was tracked down to a C0EDCU issue. Jon told me that the Hornet CC1 pressure gauge channel was renamed to . C1:Vac-CC1_pressure, and I made the change in the C0EDCU file. However, it returns a value of 9990000000.0, which the frame writer is not happy about... Keeping the old channel name makes the frame-writer run again (although the actual data is bunk).

Update 1755:

J Hanks suggested adding a 1 second offset to the daqdrc config files. This has now fixed the 0x4000 errors, and we are back to the "nominal" RTCDS status screen now - Attachment #2.

I started a GPS timing monitor script, /opt/rtcds/caltech/c1/Git/40m/scripts/cds/tempusTimingMon/tempusTimingMon.py, which runs inside a docker container on optimus. It accesses the GPS receiver (EndRun Tempus LX) status via pysnmp, and creates the following epics channels with pcaspy:

• C1:DAQ-GPS_TFOM “The Time Figure of Merit (TFOM) value ranges from 3 to 9 and indicates the current estimate of the worst case time error. It is a logarithmic scale, with each increment indicating a tenfold increase in the worst case time error boundaries.” (nominally 3, corresponding to 100 nsec)
• C1:DAQ-GPS_STATE “Current GPS signal processor state. One of 0, 1 or 2, with 0 being the acquisition state and 2 the fully locked on state.”
• C1:DAQ-GPS_SATS “Current number of GPS satellites being tracked.”
• C1:DAQ-GPS_DAC “Current 16 bit, Voltage Controlled TCXO DAC value. Typical range is 20000 to 40000, where more positive numbers have the effect of raising the TCXO frequency.”
• C1:DAQ-GPS_SNR “The current average carrier to noise ratio of all tracked satellites, in units of dB. Values less than 35 indicate weak signal conditions.”

Attached is a 1-day trend of the above channels, along with the IRIG-B timing offsets from the IOPs. No big timing excursions or slippages were seen yet, although the c1sus IOP (FEC-20) timing seems to be hopping around by one IOP sample time (15 µsec).

While recovering from the power outage, I took the opportunity to try having the timing system trigger on the rising edge of the 1 PPS signal from the GPS receiver (rather than the falling edge, as it had been doing since before the CDS upgrade). This was done in order to eliminate a timing offset between the clock used by the ADCs and the IRIG-B timestamps from the Spectracom boards. It was successful in that regard, and cleared the timing errors seen in the IOP statewords on most of the front ends.

The two exceptions were c1lsc and c1sus, where something is broken with the DuoTone readbacks. The attached plot shows the DuoTone signal in the last channel of ADC 0 on c1x01 (working normally) vs c1x02 and c1x04 (busted).

However, this change apparently also had some unexplained effect on the way the c1sbr model runs. There were frequent CPU time overruns and IPC errors. So on Sunday afternoon I reverted to the falling-edge trigger. This caused the timing errors to return, but allowed c1sbr to run normally.

Located in /opt/rtcds/caltech/c1/userapps/release/cds/common/scripts

Script 1: Diagreset.sh
Hits the diag reset buttons on the models on c1lsc and c1sus computers.

Script 2: Burtrest.sh
Restores the burt files from "today" 4am. Use a text editor if you want to change the times.

[Jenne, Kyung Ha]

We successfully suspended the 4 eddy current dampers for the first Tip Tilt.  We had some lessons learned, including how to carefully get an allen wrench in between the dampers to do up some of the screws, and how to be careful not to bend the wire while tightening the screws.  More tomorrow...

 We also wrote down the serial numbers (top center of each TT, inscribed by hand) for what tip tilt is installed where.  I then went through the elog to determine which TT was suspended with what kind of wire (thick or thin).  Summary: all installed tip tilts have thick wire, 0.0036" diameter.

As noted in elog 3295, we had found that there was similar hysteresis whether we used the thick or the thin wire, so we had decided not to go back and re-suspend every optic.

Also, since we will redo the pitch balance tomorrow with the new hardware tomorrow, I think we should put in the new LaserOptik mirrors at the same time.  We have not yet gotten phase maps of them, but we might as well do this rebalancing once, rather than twice.

Jamie and I spent some time with tip tilt SN001 this afternoon.  This was installed as SR3, so I was going to put a new LaserOptik mirror in there.  I accidentally snapped one of the wires (I forgot how strong the magnets are - one zipped from the mirror holder and captured the wire).  Jamie and I put the new LaserOptik mirror in, with the wedge correct, but we need to re-resuspend it with the 0.0036" wire tomorrow.  We'll also keep working on re-pitch aligning the other optics.

PR2 needs to be put back as a G&H, and we need to put a LaserOptik mirror into PR3.

 We resuspended SN001 this morning with 0.0036" wire.  We did as Koji suggested, and flipped the wire clamp so the suspension point is a little higher, so we'll see if that helps.  We put LaserOptik mirror SN1 into this TT001.

We put the G&H mirror back into TT004, which is PR2.  We also put a LaserOptik mirror (SN5) into TT005, which is SR3.

Jamie is working on re-pitch aligning TT004 and TT005 (we already did 001), then we can re-install them in the vacuum system later this afternoon.

 The tip tilts have all been pitch-adjusted now, and they have all been put back onto the tables, with the same serial numbers in the same places as we took them out.  Jamie also re-leveled the BS table.

Raji and I will align things after I finish measuring the MC spot positions.

I have made a preliminary sketch of the cabling involved in connecting the Tip-tilt coil drivers.   This is a preliminary document. 

2 questions (so far) regarding your diagram / doc:

We are using 3 of the feed-throughs on the BS chamber, and 1 on the OMC chamber, even though we have 2 TTs on the BS table, 1 on the OMC table, and 1 on the IMC table? Just wanted to check.

Does your list / table at the bottom include all of the cables we already have, as well as the ones we need?  (Or maybe we just have nothing so far, so this is a moot question).

 The scheme currently is to run a 25pin Kapton strip cable from BS to IMC table inside the chamber.  However we cannot do the same for the OMC table since it will cross the bellows which we often remove.  So we need to supply the one tip-tilt on the OMC table from outside.  I could not spot a spare unused feedthough on the OMC chamber.  We will have to swap one of the blank flanges for one with a few feed throughs.

We do not have any of the cables.   So everything listed has to be arranged for.   The pics are from the existing coil driver system on the SUS machine.

Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.

Recent History

The lower blades which I had given to the Physics Workshop for making a vacuum relief hole (using a sinker-EDM process) came back about ten days ago.   Merih Eken <meken@caltech.edu>,  the supervisor at the Physics Dept workshop, handled this matter for us.  The blades were sent to a local EDM machineshop and returned in about three working days ( a weekend intervened). 

Bob cleaned and handed them over to me yesterday evening.  

Current status

Today I have reassembled the four tip-tilts.  I have repacked them in clean bags (double bagged) shifted them to Clean Optics Cabinet (near the ETMX chamber).  The four tip-tilts are in the bottom-most shelf in the cabinet.  There are also some tip-tilt spares in a separate envelope.

Note:  The mirror holder is now held tightly by the eddy current dampers.  This was done for safety of the wires during transportation from LHO.  The eddy current damper in the front of the mirror has to be retracted to allow the mirror holder to swing free.  It has be to about 1mm away from the suspended mirror holder

Work Remaining

1) We need to install the quadrapus cables.  The connector placement on the BOSEM side will have some issues.  It is best to loosen the BOSEM seating as well as the connector seating screws and then push the cable connector into place.  Caution:  when the connector seating screws on the BOSEM are loosened the flexible ckt could be damaged by the loose connector.

2) Insert the mirrors into the mirror holders and balance the suspension such that the mirror's hang vertical and do not have a large yaw offset.

3) Adjust the wire suspension point height so that the flags are in the center of the BOSEM aperture.  Else they will strike against the

4) We need to adjust the position of the BOSEMs such that the shadow sensor signals are at 50%.  This ensures that all the magnets hang at an appropriate distance from their respective coils.

5) To do (3) we need to set up a shadow sensor read-out set-up for one tip-tilt (four sensors)

Final Summary of changes to mirror holder in Tip-Tilt holder.

Determining minimum range for Side Clamp:

1. The initial distance b/w wire-release point and mirror assembly COM = 0.265 mm

2. But this distance is assuming that wire-release point is at mid-point of clamp. So I'm settling on a range of +/- 1mm. The screenshots below confirm range of ~1mm between (1) side screw & protrusion and (2) clamp screw and clamp.

Determining length of tilt-weight assembly rod at the bottom to get 20mRad range

The tilt-weight assembly is made from following Mcmaster parts:
Rod   - 95412A864 18-8 SS  #2-56 Threaded Rod
Nuts  - 91855A103 18-8 SS #2-56 Acorn Cap Nut

Since the weights are fixed, only rod length can be changed to get the angle range.

So a range of 1 mm between nut's inner face and mirror-holder face should be enough. Since holder is 12 mm thick, rod length = 12mm + 2 x 1mm + 2 x (nut length) = 12 + 2 + 9.6 = 23.6 mm = 0.93 inch. So a 1" rod from Mcmaster will be fine.