 Wed Jul 28 12:00:35 2021, Yehonathan, Update, BHD, SOS assembly
|
After receiving two new tubes of EP-30 I resumed the gluing activities. I made a spreadsheet
to track the assemblies that have been made, their position on the metal sheet in the cleanroom, their magnetic field, and the batch number.
I made another batch of 6 magnets yesterday (4th batch), the assembly from the 2nd batch is currently being tested for bonding strength. |
Wed Jul 28 12:47:52 2021, Yehonathan, Update, CDS, Opto-isolator for c1auxey 
|
To simulate a differential output I used two power supplies connected in series. The outer connectors were used as the outputs and the common connector
was connected to the ground and used as a reference. I hooked these outputs to one of the differential analog channels and measured it over time using
Striptool. The setup is shown in attachment 3. |
|
Wed Jul 28 17:10:24 2021, Anchal, Update, LSC, Schnupp asymmetry
|
[Anchal, Paco]
I redid the measurement of Schnupp asymmetry today and found it to be 3.8 cm 0.9 cm. |
 Wed Jul 28 20:20:09 2021, Yehonathan, Update, General, The temperature sensors and function generator have arrived in the lab
|
I put the temperature sensors box on Anchal's table (attachment 1) and the function generator on the table in front of the c1auxey Acromag chassis
(attachment 2).
|
|
Thu Jul 29 14:51:39 2021, Paco, Update, Optical Levers, Recenter OpLevs
|
[yehonathan, anchal, paco]
Yesterday around 9:30 pm, we centered the BS, ITMY, ETMY, ITMX and ETMX oplevs (in that order) in their respective QPDs by turning the last mirror
before the QPDs. We did this after running the ASS dither for the XARM/YARM configurations to use as the alignment reference. We did this in preparation |
|
Mon Aug 2 16:18:23 2021, Paco, Update, ASC, AS WFS MICH commissioning
|
[anchal, paco]
We picked up AS WFS comissioning for daytime work as suggested by gautam. In the end we want to comission this for the PRFPMI, but also for PRMI,
and MICH for completeness. MICH is the simplest so we are starting here. |
|
Tue Aug 3 20:20:08 2021, Anchal, Update, Optical Levers, Recentered ETMX, ITMX and ETMY oplevs at good state
|
Late elog. Original time 08/02/2021 21:00.
I locked both arms and ran ASS to reach to optimum alignment. ETMY PIT > 10urad, ITMX P > 10urad and ETMX P < -10urad. Everything else
was ok absolute value less than 10urad. I recentered these three. |
|
Wed Aug 4 18:19:26 2021, paco, Update, General, Added infrasensing temperature unit to martian network
|
[ian, anchal, paco]
We hooked up the infrasensing unit to power and changed its default IP address from 192.168.11.160 (factory default) to 192.168.113.240 in the
martian network. The sensor is online with user controls and the usual password for most workstations in that IP address. |
|
Thu Aug 5 14:59:31 2021, Anchal, Update, General, Added temperature sensors at Yend and Vertex too
|
I've added the other two temperature sensor modules on Y end (on 1Y4, IP: 192.168.113.241) and in the vertex on (1X2, IP: 192.168.113.242). I've
updated the martian host table accordingly. From inside martian network, one can go
to the browser and go to the IP address to see the temperature sensor status . These sensors can be set to trigger alarm and send emails/sms etc if temperature |
|
Fri Aug 6 13:13:28 2021, Anchal, Update, BHD, c1teststand subnetwork now accessible remotely
|
c1teststand subnetwork is now accessible remotely. To log into this network, one needs to do following:
Log into nodus or pianosa. (This will only work from these two computers)
ssh -CY
controls@192.168.113.245
Password is our usual workstation password.
This will log you into c1teststand network.
From |
|
Fri Aug 6 17:10:19 2021, Paco, Update, IMC, MC rollercoaster
|
[anchal, yehonatan, paco]
For whatever reason (i.e. we don't really know) the MC unlocked into a weird state at ~ 10:40 AM today. We first tried to find a likely cause
as we saw it couldn't recover itself after ~ 40 min... so we decided to try a few things. First we verified that no suspensions were acting |
|
Mon Aug 9 10:38:48 2021, Anchal, Update, BHD, c1teststand subnetwork now accessible remotely
|
I had to add following two lines in the /etc/network/interface file to make the special ip routes persistent even after reboot:
post-up ip route add 192.168.113.200 via 10.0.1.1 dev eno1
post-up ip route add 192.168.113.216 via 10.0.1.1 dev eno1 |
|
Tue Aug 10 17:24:26 2021, paco, Update, General, Five day trend
|
Attachment 1 shows a five and a half day minute-trend of the three temperature sensors. Logging started last Thursday ~ 2 pm when all sensors were finally
deployed. While it appears that there is a 7 degree gradient along the XARM it seems like the "vertex" (more like ITMX) sensor was just placed
on top of a network switch (which feels lukewarm to the touch) so this needs to be fixed. A similar situation is observed in the ETMY sensor. I shall do |
|
Wed Aug 11 11:35:36 2021, Paco, Update, LSC, PRMI MICH orthogonality plan
|
[yehonathan, paco]
Yesterday we discussed a bit about working on the PRMI sensing matrix.
In particular we will start with the "issue" of non-orthogonality in the MICH actuated by BS + PRM. Yesterday afternoon we played a |
|
Wed Aug 11 12:06:40 2021, Yehonathan, Update, CDS, Opto-isolator for c1auxey
|
I redid the differential input experiment using the DS360 function generator we recently got. I generated a low frequency (0.1Hz) sine wave signal with
an amplitude 0.5V and connected the + and - output to a differential input on the new c1auxcey Acromag chassis. I recorded a time series of the corresponding
EPICS channel with and without the common on the DS360 connected to the Ref connector on the Acromag unit. The common connector on the DS360 is not normally |
|
Thu Aug 12 11:04:27 2021, Paco, Update, General, PSL shutter was closed this morning
|
Thu Aug 12 11:04:42 2021 Arrived to find the PSL shutter closed. Why? Who? When? How? No elog, no fun. I opened it, IMC is now locked, and the arms were
restored and aligned. |
|
Thu Aug 12 14:59:25 2021, Koji, Update, General, PSL shutter was closed this morning
|
| What I was afraid of was the vacuum interlock. And indeed there was a pressure surge this morning. Is this real? Why didn't we receive the alert? |
|
Thu Aug 12 20:52:04 2021, Koji, Update, General, PSL shutter was closed this morning
|
I did a bit more investigation on this.
- I checked P1~P4, PTP2/3, N2, TP2, TP3. But found only P1a and P2 were affected.
- Looking at the min/mean/max of P1a and P2 (Attachment 1), the signal had a large fluctuation. It is impossible to have P1a from 0.004 to 0 |
|
Mon Aug 16 23:30:34 2021, Paco, Update, CDS, AS WFS commissioning; restarting models
|
[koji, ian, tega, paco]
With the remote/local assistance of Tega/Ian last friday I made changes on the c1sus model by connecting the C1:ASC model outputs (found within
a block in c1ioo) to the BS and PRM suspension inputs (pitch and yaw). Then, Koji reviewed these changes today and made me notice that no changes are actually |
|
Tue Aug 17 04:30:35 2021, Koji, Update, SUS, New electronics
|
Received:
Aug 17, 2021 2x ISC Whitening
Delivered 2x Sat Amp board to Todd |
|
Wed Aug 18 20:30:12 2021, Anchal, Update, ASS, Fixed runASS scripts
|
Late elog: Original time of work Tue Aug 17 20:30 2021
I locked the arms yesterday remotely and tried running runASS.py scripts (generally ran by clicking Run ASS buttons on IFO OVERVIEW screen
of ASC screen). We have known for few weeks that this script stopped working for some reason. It would start the dithering and would optimize the alignment |
|
Thu Aug 19 03:23:00 2021, Anchal, Update, CDS, Time synchornization not running
|
I tried to read a bit and understand the NTP synchronization implementation in FE computers. I'm quite sure that NTP synchronization should be 'yes'
if timesyncd are running correctly in the output of timedatectl in these computers. As Koji reported in 15791,
this is not the case. I logged into c1lsc, c1sus and c1ioo and saw that RTC has drifted from the software clocks too which does not happen if NTP synchronization |
|
Thu Aug 19 14:14:49 2021, Koji, Update, CDS, Time synchornization not running
|
131.215.239.14 looks like Caltech's NTP server (ntp-02.caltech.edu)
https://webmagellan.com/explore/caltech.edu/28415b58-837f-4b46-a134-54f4b81bee53
|
|
Fri Aug 20 00:28:55 2021, Anchal, Update, CDS, Time synchornization not running
|
I added ntpserver as a known host name for address 192.168.113.201 (fb1's address where ntp server is running) in the martian host list in the following
files in Chiara:
/var/lib/bind/martian.hosts
/var/lib/bind/rev.113.168.192.in-addr.arpa
Note: |
|
Fri Aug 20 06:24:18 2021, Anchal, Update, CDS, Time synchornization not running
|
I read on some stack exchange that 'NTP synchornized' indicator turns 'yes' in the output of command timedatectl only when RTC clock
has been adjusted at some point. I also read that timesyncd does not do the change if the time difference is too much, roughly more than 3 seconds.
So I logged into all FE machines and ran sudo hwclock -w to synchronize them all to the system |
|
Mon Aug 23 11:51:26 2021, Koji, Update, General, Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am
|
Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am (more like 9:34am according to nodus
log)
nodus: rebooted. ELOG/apache/svn is running. (looks like Anchal worked on it) |
|
Mon Aug 23 15:25:59 2021, Ian MacMillan, Update, CDS, SUS simPlant model
|
I am adding a State-space block to the SimPlant cds model using the example Chris gave. I made a new folder in controls called SimPlantStateSpace. wI
used the code below to make a state-space LTI model with a 1D pendulum then I converted it to a discrete system using c2d
matlab function. Then I used these in the rtss.m file to create the state space code I need in the SimPlantStateSpace_1D_model.h |
|
Mon Aug 23 19:00:05 2021, Koji, Update, General, Campus Wide Power Glitch Reported: Monday, 8/23/21 at 9:30am
|
Restarting the RTS was unsuccessful because of the timing discrepancy error between the RT machines and the FB. This time no matter how we try to set
the time, the IOPs do not run with "DC status" green. (We kept having 0x4000)
We then decided to work on the recovery without the data recorded. After some burtrestores, the IMC was locked and the spot appeared on the AS |
|
Mon Aug 23 22:51:44 2021, Anchal, Update, General, Time synchronization efforts
|
Related elog thread: 16286
I didn't really achieve anything but I'm listing what I've tried.
I know now that the timesyncd isn't working because systemd-timesyncd is known to have issues when running on a read-only file system. |
|
Tue Aug 24 09:22:48 2021, Anchal, Update, General, Time synchronization working now
|
Jamie told me to use chroot to log in into the chroot jail of debian os that are exported for the FEs and install ntp there. I took following steps at
the end of which, all FEs have NTP synchronized now.
I logged into fb1 through nodus.
chroot /diskless/root.jessie /bin/bash took |
|
Tue Aug 24 18:11:27 2021, Paco, Update, General, Time synchronization not really working
|
tl;dr: NTP servers and clients were never synchronized, are not synchronizing even with ntp... nodus is synchronized but uses chronyd; should
we use chronyd everywhere?
Spent some time investigating the ntp synchronization. In the morning, after Anchal set up all the ntp servers / FE clients I tried restarting |
|
Tue Aug 24 18:44:03 2021, Koji, Update, CDS, FB is writing the frames with a year old date
|
Dan Kozak pointed out that the new frame files of the 40m has not been written in 2021 GPS time but 2020 GPS time.
Current GPS time is 1313890914 (or something like that), but the new files are written as C-R-1282268576-16.gwf
I don't know how this can happen but this may explain why we can't have the agreement between the FB gps time and the RTS gps time. |
|
Tue Aug 24 22:37:40 2021, Anchal, Update, General, Time synchronization not really working
|
I attempted to install chrony and run it on one of the FE machines. It didn't work and in doing so, I lost the working NTP client service on the
FE computers as well. Following are some details:
I added the following two mirrors in the apt source list of root.jessie at /etc/apt/sources.list |
|
Wed Aug 25 08:53:33 2021, Jordan, Update, SUS, 2" Adapter Ring for SOS Arrived 8/24/21
|
8 of the 2"->3" adapter rings (D2100377) arrived from RDL yesterday. I have not tested the threads but dimensional inspection on SN008 cleared.
Parts look very good. The rest of the parts should be shipping out in the next week. |
|
Wed Aug 25 11:48:48 2021, Yehonathan, Update, CDS, c1auxey assembly
|
After confirming that, indeed, leaving the RTN connection floating can cause reliability issues we decided to make these connections in the c1auxex analog
input units.
According to Johannes' wiring scheme (excluding |
|
Wed Aug 25 17:31:30 2021, Paco, Update, CDS, FB is writing the frames with a year old date
|
[paco, tega, koji]
After invaluable assistance from Jamie in fixing this yearly offset in the gps time reported by cat /proc/gps, we managed to restart the real
time system correctly (while still manually synchronizing the front end machine times). After this, we recovered the mode cleaner and were able to lock |
|
Wed Aug 25 18:20:21 2021, Jamie, Update, CDS, GPS time on fb1 fixed, dadq writing correct frames again
|
I have no idea what happened to the GPS timing on fb1, but it seems like the issue was coincident with the power
glitch on Monday.
As was noted by Koji above, the GPS time kernel interface was off by a year, which was causing the frame builder to write out files with the |
|
Thu Aug 26 10:10:44 2021, Paco, Update, CDS, FB is writing the frames with a year old date
|
[paco, ]
We went over the X end to check what was going on with the TRX signal. We spotted the ground terminal coming from the QPD is loosely touching
the handle of one of the computers on the rack. When we detached it completely from the rack the noise was gone (attachment 1). |
|
Wed Sep 1 14:16:21 2021, Jordan, Update, VAC, Empty N2 Tanks
|
The right N2 tank had a bad/loose valve and did not fully open. This morning the left tank was just about empty and the right tank showed 2000+ psi on
the gauge. Once the changeover happened the copper line emptied but the valve to the N2 tank was not fully opened. I noticed the gauges were both reading
zero at ~1pm just before the meeting. I swapped the left tank, but not in time. The vacuum interlocks tripped at 1:04 pm today when the N2 pressure to |
|
Thu Sep 2 19:28:02 2021, Koji, Update, , This week's FB1 GPS Timing Issue Solved
|
After the disk system trouble, we could not make the RTS running at the nominal state. A part of the troubleshoot FB1 was rebooted. But the we found
that the GPS time was a year off from the current time
controls@fb1:/diskless/root/etc 0$ cat /proc/gps |
|
Thu Sep 2 19:47:38 2021, Koji, Update, CDS, This week's FB1 GPS Timing Issue Solved
|
After the reboot daqd_dc was not working, but manual starting of open-mx / mx services solved the issue.
sudo systemctl start open-mx.service
sudo systemctl start mx.service |
|
Thu Sep 2 20:44:18 2021, Koji, Update, CDS, Chiara DHCP restarted
|
We had the issue of the RT machines rebooting. Once we hooked up the display on c1iscex, it turned out that the IP was not given at it's booting-up.
I went to chiara and confirmed that the DHCP service was not running
~>sudo service isc-dhcp-server status |
|
Thu Sep 2 20:47:19 2021, Koji, Update, CDS, Chiara DHCP restarted
|
[Paco, Tega, Koji]
Once chiara's DHCP is back, things got much more straight forward.
c1iscex and c1iscey were rebooted and the IOPs were launched without any hesitation. |
|
Wed Sep 8 18:00:01 2021, Koji, Update, VAC, cronjobs & N2 pressure alert
|
In the weekly meeting, Jordan pointed out that we didn't receive the alert for the low N2 pressure.
To check the situation, I went around the machines and summarized the cronjob situation.
[40m wiki: cronjob summary] |
|
Wed Sep 8 19:06:14 2021, Koji, Update, General, Backup situation
|
Tega mentioned in the meeting that it could be safer to separate some of nodus's functions from the martian file system.
That's an interesting thought. The summary pages and other web services are linked to the user dir. This has high traffic and can cause the
issure of the internal network once we crash the disk. |
|
Mon Sep 13 04:12:01 2021, Tega, Update, General, Added temperature sensors at Yend and Vertex too
|
I finally got the modbus part working on chiara, so we can now view the temperature data on any machine on the martian network, see Attachment
1.
I also updated the entries on /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini, as suggested by Koji, to include the SensorGatway temperature |
|
Mon Sep 13 09:15:15 2021, Paco, Update, LSC, MC unlocked?
|
Came in at ~ 9 PT this morning to find the IFO "down". The IMC had lost its lock ~ 6 hours before, so at about 03:00 AM. Nothing seemed like
the obvious cause; there was no record of increased seismic activity, all suspensions were damped and no watchdog had tripped, and the pressure trends
similar to those in recent pressure incidents show nominal behavior (Attachment #1). |
|
Mon Sep 13 14:32:25 2021, Yehonathan, Update, CDS, c1auxey assembly
|
So we agreed that the RTNs points on the c1auxex Acromag chassis should just be grounded to the local Acromag ground as it just needs a stable reference.
Normally, the RTNs are not connected to any ground so there is should be no danger of forming ground loops by doing that. It is probably best to use the
common wire from the 15V power supplies since it also powers the VME crate. I took the spectra of the ETMX OSEMs (attachment) for reference and proceeding |
|
Mon Sep 13 15:14:36 2021, Anchal, Update, LSC, Xend Green laser injection mirrors M1 and M2 not responsive
|
I was showing some green laser locking to Tega, I noticed that changing the PZT sliders of M1/M2 angular position on Xend had no effect on locked TEM01
or TEM00 mode. This is odd as changing these sliders should increase or decrease the mode-matching of these modes. I suspect that the controls are not
working correctly and the PZTs are either not powered up or not connected. We'll investigate this in near future as per priority. |
|
Mon Sep 13 18:19:25 2021, Tega, Update, Computer Scripts / Programs, Moved modbus service from chiara to c1susaux
|
[Tega, Anchal, Paco]
After talking to Anchal, it was made clear that chiara is not the place to host the modbus service for the temperature sensors. The obvious machine
is c1pem, but the startup cmd script loads c object files and it is not clear how easy it would integrate the modbus functionality since we can only login |