After making the CDS modification, CESAR was tested with ALS.

First of all, we run CESAR with threshold of 10. This means that the error signal always used ALS.
The ALS was scanned over the resonance. The plot of the scan can be found in EricQ's elog.
At each point of the scan, the arm stability is limited by the ALS.

Using this scan data, we could adjust the gains for the TR and PDH signals. Once the gains were adjusted
the threshold was lowered to 0.25. This activates dynamic signal blending.

ALS was stabilized with XARM FM1/2/3/5/6/7/9. The resonance was scanned. No glitch was observed.
This is some level of success already.

Next step was to fully hand off the control to PDH. But this was not successfull. Everytime the gain for the TR was
reduced to zero, the lock was lost. When the TR is removed from the control, the raw PDH signal is used fot the control
without normalization. Without turning on FM4, we lose 60dB of DC gain. Therefore the residual motion may have been
too big for the linear range of the PDH signal. This could be mitigated by the normalization of the PDH signal by the TR.

Today we modified the CESAR block.

- Now the sign(X) function is in a block.

- We decided to use the linearization of the PDH.

- By adding the offset to the TR signal used for the switching between TR and PDH, we can force pure 1/sqrt(TR) or pure PDH to control the cavity.

[Nic, Jenne, EricQ, and Koji]

We have used CESAR successfully to bring the Xarm into resonance.  We start with the ALS signal, then as we approach resonance, the error signal is automatically transitioned to 1/sqrt(TRX), and ramped from there to POX, which we use as the PDH signal.

In the first plot, we have several spectra of the CESAR output signal (which is the error signal for the Xarm), at different arm resonance conditions.  Dark blue is the signal when we are locked with the ALS beatnote, far from IR resonance.  Gold is when we are starting to see IR resonance (arm buildup of about 0.03 or more), and we are using the 1/sqrt(TRX) signal for locking.  Cyan is after we have achieved resonance, and are using only the POX PDH signal.  Purple is the same condition as cyan, except that we have also engaged the low frequency boosts (FM 2, 3, 4) in the locking servo.  FM4 is only usable once you are at IR resonance, and locked using the PDH signal.  We see in the plot that our high frequency noise (and total RMS) decreases with each stage of CESAR (ALS, 1/sqrt(TR) and PDH). 

To actually achieve the gold noise level of 1/sqrt(TR), we first had to increase the analog gain by swapping out a resistor on the whitening board. 

The other plots attached are time series data.  For the python plots (last 2), the error signals are calibrated to nanometers, but the dark blue, which is the transmitted power of the cavity, is left in normalized power units (where 1 is full IR resonance). 

In the scan from off resonance to on resonance, around the 58 second mark, we see a glitch when we engage FM4, the strong low frequency boosts.  Around the 75 second mark we turned off any contribution from 1/sqrt(TR), so the noise decreases once we are on pure PDH signal. 

In the scan through the resonance, we see a little more clearly the glitch that happens when we switch from ALS to IR signals, around the 7 and 12 second marks. 

We want to make some changes, so that the transition from ALS to IR signals is more smooth, and not a discrete switch.

[Jenne, EricQ]

As Koji suggested in his email this afternoon, we looked at how much actuator range is required for various forms of arm locking:  (1) "regular" PDH lock aquisition, (2) ALS lock acquisition, (3) CESAR cooling.

To start, I looked at the spectra and time series data of the control signal (XARM_OUT) for several locking situations.  Happily, when the arm is at the half fringe, where we expect the 1/sqrt(TRX) signal to be the most sensitive (versus the same signal at other arm powers), we see that it is in fact more quiet than even the PDH signal.  Of course, we can't ever use this signal once the arm is at resonance, so we haven't discovered anything new here.

EricQ then made some violin plots with the time series data from these situations, and we determined that a limit of ~400 counts encompasses most of the steady-state peak-to-peak output from locking on the PDH signal.

[ericq: What's being plotted here are "kernel density estimates" of the time series data of XARM_OUT when locked on these signals. The extent of the line goes to the furthest outlier, while the dashed and dotted lines indicate the median and quartiles, respectively]

I tried acquiring ALS and transitioning to final PDH signals with different limiters set in the Xarm servo.  I discovered that it's not too hard to do with a limit of 400 counts, but that below ~350 counts, I can't keep the ALS locked for long enough to find the IR resonance.  Here's a plot of acquiring ALS lock, scanning for the resonance, and then using CESAR to transition to PDH, with the limit of 400 counts in place, and then a lockloss at the end.  Even though I'm hitting the rails pretty consistently, until I transition to the more quiet signals, I don't ever lose lock (until, at the end, I started pushing other buttons...).

After that, I tried acquiring lock using our "regular" PDH method, and found that it wasn't too hard to capture lock with a limit of 400, but with limits below that I can't hold the lock through the boosts turning on.

Finally, I took spectra of the XARM_OUT control signal while locked using ALS only, but with different limiter values. Interestingly, I see much higher noise between 30-300 Hz with the limiter engaged, but the high frequency noise goes down.  Since the high frequency is dominating the RMS, we see that the RMS value is actually decreasing a bit (although not much).

We have not made any changes to the LSC model, so there is still no smoothing between the ALS and IR signals.  That is still on the to-do list.  I started modifying things to be compatible with CARM rather than a single arm, but that's more of a daytime-y task, so that version of the c1lsc model is saved under a different name, and the model that is currently compiled and running is reverted as the "c1lsc.mdl" file.

Asymptotic cooling of the mirror motion with CESAR was tested.

With ALS and the full control bandwidth (UGF 80-100Hz), the actuator amplitude of 8000cnt_pp is required.

Varying control bandwidth depending on the noise level of the signal, the arm was brought to the final configuration with the actuator amplitude of 800cnt_pp.

Current computer status:

All fast machines except c1iscey are up and running. I can't ssh to c1iscey, so I'll need to go down to the end station and have a look-see. On the c1lsc machine, neither the c1oaf nor the c1cal models are running (but for the oaf model, we know that this is because we need to revert the blrms block changes to some earlier version, see Jamie's elog 9911).

Daqd process is running on framebuilder.  However, when I try to open dataviewer, I get the popup error saying "Can't connect to rb", as well as an error in the terminal window that said something like "Error getting chan info".

Slow machines c1psl, c1auxex and c1auxey are not running (can't telnet to them, and white boxes on related medm screens for slow channels).  All other slow machines seem to be running, however nothing has been done to them to point them at the new location of the shared hard drive, so their status isn't ready to green-light yet.

Things that we did on Friday for the fast machines:

The shared hard drive is "physically" on Chiara, at /home/cds/.  Links are in place so that it looks like it's at the same place that it used to be:  /opt/rtcds/...... 

The first nameserver on all of the workstation machines inside of the file /etc/resolv.conf has been changed to be 192.168.113.104, which is Chiara's IP address (it used to be 192.168.113.20, which was linux1).  This change has also been made on the framebuilder, and in the framebuilder's /diskless/root/etc/resolv.conf file, which is what all of the fast front ends look to. 

On the framebuilder, and in the /diskless place for the fast front ends, presumably we must have changed something to point at the new location for the shared drive, but I don't remember how we did that [ERIC, what did we do???]

The slow front ends that we have tried changing have not worked out. 

First, we tried plugging a keyboard and monitor into c1auxey.  When we key the crate to reboot the machine, we get some error message about a "disk A drive error", but then it goes on to prompt pushing F1 for something, and F2 for entering setup.  No matter what we press, nothing happens.  c1auxey is still not running.

We were able to telnet into c1auxex, c1psl, and c1iool0.  On each of those machines, at the prompt, we used the command "bootChange".  This initially gives us a series of:

$ telnet c1susaux
Trying 192.168.113.55...
Connected to c1susaux.
Escape character is '^]'.

c1susaux > bootChange

'.' = clear field;  '-' = go to previous field;  ^D = quit

boot device          : ei
processor number     : 0
host name            : linux1
file name            : /cvs/cds/vw/mv162-262-16M/vxWorks
inet on ethernet (e) : 192.168.113.55:ffffff00
inet on backplane (b):
host inet (h)        : 192.168.113.20
gateway inet (g)     :
user (u)             : controls
ftp password (pw) (blank = use rsh):
flags (f)            : 0x0
target name (tn)     : c1susaux
startup script (s)   : /cvs/cds/caltech/target/c1susaux/startup.cmd
other (o)            :

value = 0 = 0x0
c1susaux >

If we go through that again (it comes up line-by-line, and you must press Enter to go to the next line) and put a period a the end of the Host Name line, and the Host Inet (h) line, they will come up blank the next time around.  So, the next time you run bootChange, you can type "chiara" for the host name, and "192.168.113.104" for the "host inet (h)".  If you run bootChange one more time, you'll see that the new things are in there, so that's good.

However, when we then try to reboot the computer, I think the machines weren't coming back after this point.  (Unfortunately, this is one of those things that I should have elogged back on Friday, since I don't remember precisely).  Certainly whatever the effect was, it wasn't what I wanted, and I left with the machines that I had tried rebooting, not running.

In all of the fstabs, we're using chiara's IP instead of name, so that if the nameserver part isn't working, we can still get the NFS mounts.

On control room computers, we mount the NFS through /etc/fstab having lines like:

192.168.113.104:/home/cds /cvs/cds nfs rw,bg 0 0
fb:/frames /frames nfs ro,bg 0 0

Then, things like /cvs/cds/foo are locally symlinked to /opt/foo

For the diskless machines, we edited the files in /diskless/root. On FB, /diskless/root/etc/fstab becomes

master:/diskless/root                   /         nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
master:/usr                             /usr      nfs     sync,hard,intr,ro,nolock,rsize=8192,wsize=8192    0 0
master:/home                            /home     nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
none                                    /proc     proc    defaults          0 0
none                                    /var/log        tmpfs   size=100m,rw    0 0
none                                    /var/lib/init.d tmpfs   size=100m,rw    0 0
none                                    /dev/pts        devpts  rw,nosuid,noexec,relatime,gid=5,mode=620        0 0
none                                    /sys            sysfs   defaults        0 0
master:/opt                             /opt      nfs    async,hard,intr,rw,nolock  0 0
192.168.113.104:/home/cds/rtcds         /opt/rtcds      nfs     nolock  0 0
192.168.113.104:/home/cds/rtapps        /opt/rtapps     nfs     nolock  0 0

("master" is defined in /diskless/root/etc/hosts to be 192.168.113.202, which is fb's IP)

and /diskless/root/etc/resolv.conf becomes:

search martian

nameserver 192.168.113.104 #Chiara

After the Great Computer Meltdown of 2014, we forgot about poor c0rga, which is why the RGA hasn't been recording scans for the past several months (as Steve noted in elog 10548).

Q helped me remember how to fix it.  We added 3 lines to its /etc/fstab file, so that it knows to mount from Chiara and not Linux1.  We changed the resolv.conf file, and Q made some simlinks.

Steve and I ran ..../scripts/RGA/RGAset.py on c0rga to setup the RGA's settings after the power outage, and we're checking to make sure that the RGA will run right now, then we'll set it back to the usual daily 4am run via cron.

EDIT, JCD:  Ran ..../scripts/RGA/RGAlogger.py, saw that it works and logs data again.  Also, c0rga had a slightly off time, so I ran sudo ntpdate -b -s -u pool.ntp.org, and that fixed it.


I really think it's a bad idea to be making all these names changes.  You're making things much much harder for yourselves.

Instead of repointing everything to a new host, you should have just changed the DNS to point the name "linux1" to the IP address of the new server.  That way you wouldn't need to reconfigure all of the clients.  That's the whole point of  name service: use a name so that you don't need to point to a number.

Also, pointing to an IP address for this stuff is not a good idea.  If the IP address of the server changes, everything will break again.

Just point everything to linux1, and make the DNS entries for linux1 point to the IP address of chiara.  You're doing all this work for nothing!

RXA: Of course, I understand what DNS means. I wanted to make the changes to the startup to remove any misconfigurations or spaghetti mount situations (of which we found many). The way the VME162 are designed, changing the name doesn't make the fix - it uses the number instead. And, of course, the main issue was not the DNS, but just that we had to setup RSH on the new machine. This is all detailed in the ELOG entries we've made, but it might be difficult to understand remotely if you are not familiar with the 40m CDS system.

I noticed that the fb lights on all of the models on the c1lsc machine are red, and that even though the MC was locked, there was no light flashing in the IFO. Also, all of the EPICS values on the LSC screen were frozen.

I tried restarting the ntp server on the frame builder, as in elog 9567, but that didn't fix things.  (I realized later that the symptom there was a red light on every machine, while I'm just seeing problems with c1lsc. 

I did an mxstream restart, as a harmless thing that had some small hope of helping (it didn't). 

I logged on to c1lsc, and restarted all of the models (rtcds restart all), which stops all of the models (IOP last), and then restarts them (IOP first).  This did not change the status of the lights on the status screen, but it did change the positioning of some optics (I suspect the tip tilts) significantly, and I was again seeing flashes in the arms.  The LSC master enable switch was off, so I don't think that it was trying to send any signals out to the suspensions.  The ASS model, which sends signals out to the input pointing tip tilts runs on c1lsc, and it was about when the ass model was restarted that the beam came back.  Also, there are no jumps in any of the SOS OSEM sensors in the last few hours, except me misaligning and restoring the optics.  I we don't have sensors on the tip tilts, so I can't show a jump in their positioning, but I suspect them.

I called Jamie, and he suggested restarting the machine, which I did.  (Once again, the beam went somewhere, and I saw it scattering big-time off of something in the BS chamber, as viewed on the PRM-face camera).  This made the oaf and cal models run (I think they were running before I did the restart all, but they didn't come back after that.  Now, they're running again).  Anyhow, that did not fix the problem.  For kicks, I re-ran mxstream restart, and diag reset, to no avail.  I also tried running the sudo /etc/init.d/ntp-client restart command on just the lsc machine, but it doesn't know the command 'ntp-client'. 

Jamie suggested looking at the timing card in the chassis, to ensure all of the link lights are on, etc.  I will do this next.


The LSC machine isn't any better, and now c1sus is showing the same symptoms.  Lame.

The link lights on the c1lsc I/O chassis and on the fiber timing system are the same as all other systems.  On the timing card in the chassis, the light above the fibers was solid-on, and the light below blinks at 1pps. 

Koji and I power-cycled both the lsc I/O chassis, and the computer, including removing the power cables (after softly shutting down) so there was seriously no power.  Upon plugging back in and turning everything on, no change to the timing status.  It was after this reboot that the c1sus machine also started exhibiting symptoms. 

[Jenne, Q]

Everything seems to be back up and running.

The computers weren't such a big problem (or at least didn't seem to be).  I turned off the watchdogs, and remotely rebooted all of the computers (except for c1lsc, which Manasa already had gotten working).  After this, I also ssh-ed to c1lsc and restarted all of the models, since half of them froze or something while the other computers were being power cycled. 

However, this power cycling somehow completely screwed up the vertex suspensions.  The MC suspensions were fine, and SRM was fine, but the ITMs, BS and PRM were not damping.  To get them to kind of damp rather than ring up, we had to flip the signs on the pos and pit gains.  Also, we were a little suspicious of potential channel-hopping, since touching one optic was occasionally time-coincident with another optic ringing up.  So, no hard evidence on the channel hopping, but suspicions.

Anyhow, at some point I was concerned about the suspension slow computer, since the watchdogs weren't tripping even though the osem sensor rmses were well over the thresholds, so I keyed that crate.  After this, the watchdogs tripped as expected when we enabled damping but the RMS was higher than the threshold.

I eventually remotely rebooted c1sus again.  This totally fixed everything.  We put all of the local damping gains back to the values that we found them (in particular, undoing our sign flips), and everything seems good again.  I don't know what happened, but we're back online now. 

Q notes that the bounce mode for at least ITMX (haven't checked the others) is rung up.  We should check if it is starting to go down in a few hours.

Also, the FSS slow servo was not running, we restarted it on op340m.

I'm trying to figure out what went wrong last night. But the morning status...the computers are down.

Comsol 4.3b is now installed under /cvs/cds/caltech/apps/linux64/COMSOL43b. I've left the existing Comsol 4.2 installation alone; according to the Comsol installation guide [PDF], it is unaffected by the new install. On megatron I've made a symlink so that you can call comsol in bash to start Comsol 4.3b.

The first time I ran comsol server, it asked me to choose a username/password combo, so I made it the same as the combo used to log on to megatron.

Edit: I've also added a ~/.screenrc on megatron (based on this Stackoverflow answer) so that I don't constantly go nuts trying to figure out if I'm already inside a screen session.

Comsol 4.4 is now installed under /cvs/cds/caltech/apps/linux64/COMSOL44. I've left the other installations alone. I've changed the symlink on megatron so that comsol now starts Comsol 4.4.

The first time I ran comsol server, it asked me to choose a username/password combo, so I made it the same as the combo used to log on to megatron.

We should consider uninstalling some of the older Comsol versions; right now we have 4.0, 4.2, 4.3b, and 4.4 installed.

After further astigmatism/tolerance analysis [ELOG 15380, 15387] our conclusion is that the stock-optic telescope designs [ELOG 15379] are sufficient for the first round of BHD testing. However, for the final BHD hardware we should still plan to procure the custom-curvature optics [DCC E2000296]. The optimized custom-curvature designs are much more error-tolerant and have high probability of achieving < 2% mode-matching loss. The stock-curvature designs can only guarantee about 95% mode-matching.

Below are the final distances between optics in the relay paths. The base set of distances is taken from the 2020-05-21 layout. To minimize the changes required to the CAD model, I was able to achieve near-maximum mode-matching by moving only one optic in each relay path. In the AS path, AS3 moves inwards (towards the BHDBS) by 1.06 cm. In the LO path, LO4 moves backwards (away from the BHDBS) by 3.90 cm.

AS Path

LO Path

After testing both the Conductive and Isolated front panels on the ALS delay line box using the actual beatbox and comparing this to the previous setup, I found that the conductive SMAs improved crosstalk the most. Also, as the old cables were 30m and the new ones are 50m, Eric gave me a conversion factor to apply to the new cables to normalize the comparison.

I used an amplitude of 1.41 Vpp and drove the following frequencies through each cable:
      X: 30.019 MHz          Y: 30.019203 MHz

which gave a difference of 203 Hz. 

In the first figure, it can be seen that, for the old setup with the 30m cables, in both cables there is a spike at 203 Hz with an amplitude of above 4 m/s^2/sqrt(Hz). When the 50m cables were measured in the box with the conductive front panel, the amplitude drops at 203 Hz by a factor of around 3. I also compared the isolated front panel with the old setup, and found that the isolated front panel worse by a factor of just over 2 than the old setup. Therefore, I think that using the conductive front panel for the ALS Delay Line box will reduce noise and crosstalk between the cables the most. 

I have gone through the different individual degrees of freedom on the IFO_CONFIGURE screen (I haven't done anything to the full IFO yet), and updated the burt snapshot request files to include all of the trigger thresholds (the DoF triggers were there, but the FM triggers and the FM mask - which filter modules to trigger - were not).  I also made all of the restore scripts (which does the burt restore for all those settings) the same.  They were widely different, rather than just different optics chosen for misaligning and restoring.

Before doing any of this work, I moved the whole folder ..../caltech/c1/burt/c1ifoconfigure to ..../caltech/c1/burt/c1ifoconfigure_OLD_but_SAVE , so we can go back and look at the past settings, if we need to.

I also changed the "C1save{DoF}" scripts to ask for keyboard input, and then added them as options to the CONFIGURE screen.  The keyboard input is so that people randomly pushing the buttons don't overwrite our saved burt files.  Here's the secret:  It asks if you are REALLY sure you want to save the configuration.  If you are, type the word "really", then hit enter (as in yes, I am really sure).  Any other answer, and the script will tell you that it is quitting without saving.

I have also removed the "PRM" option, since we really only want the "PRMsb" for almost all purposes.

Also, I removed access to the very, very old text file about how to lock from the screen.  That information is now on the wiki:  https://wiki-40m.ligo.caltech.edu/How_To/Lock_the_Interferometer

I have noted in the drop-down menus that the "align" functions are not yet working.  I know that Den has gotten at least one of the arms' ASSes working today, so once those scripts are ready, we can call them from the configure screen.

Anyhow, the IFO_CONFIGURE screen should be back to being useful!

Following Jamie's table in elog 8654, I have connected up the channels 0, 1 and 2 from ADC0 on the IOO computer to rfm send blocks, which send the signals over to the rfm model, and then I use dolphin send blocks to get over to the ass model on the lsc machine. 

I'm using the 1st 3 channels on the Pentek Generic interface board, which is why I'm using channels 0,1,2. 

I compiled all 3 models (ioo, rfm, ass), and restarted them.  I also restarted the daqd on the fb, since I put in a temporary set of filter banks in the ass model, to use as sinks for the signal (since I haven't done anything else to the ASS model yet).

All 3 models were checked in to the svn.

[Anchal, Paco]

I used the UPS that was providing battery backup for chiara earlier (a APS Back-UPS Pro 1000), to provide battery backup to Megatron. This completes UPS backup to all important computers in the lab. Note that this UPS nominally consumes 36% of UPS capacity in power delivery but at start-up, Megatron was many fans that use up to 90% of the capacity. So we should not use this UPS for any other computer or equipment.

While doing so, we found that PS3 on Megatron was malfunctioning. It's green LED was not lighting up on connecting to power, so we replaced it from the PS3 of old FB computer from the same rack. This solved this issue.

Another thing we found was that Megatron on restart does not get configured to correct nameserver resolution settings and loses the ability to resolve names chiara and fb1. This results in the nfs mounts to fail which in turn results in the script services to fail. We fixed this by identifying that the NetworkManager of ubuntu was not disabled and would mess up the nameserver settings which we want to be run by systemd-resolved instead. We corrected the symbolic link: /etc/resolv.conf -> /run/systemd/resolve/resolv.conf. the we stopped and diabled the NetworkManager service to keep this persistent on reboot. Following are the steps that did this:

I connected c1sus2 to the martian network by splitting the c1sim connection with a 5-way switch. I also ran another ethernet cable from the second port of c1sus2 to the DAQ network switch on 1X7.

Then I logged into chiara and added the following in chiara:/etc/dhcp/dhcpd.conf :

host c1sus2 {
  hardware ethernet 00:25:90:06:69:C2;
  fixed-address 192.168.113.92;
}

And following line in chiara:/var/lib/bind/martian.hosts :

c1sus2          A    192.168.113.92

Note that entires c1bhd is already added in these files, probably during some earlier testing by Gautam or Jon. Then I ran following to restart the dhcp server and nameserver:

~> sudo service bind9 reload
[sudo] password for controls:
 * Reloading domain name service... bind9                                                 [ OK ]
~> sudo service isc-dhcp-server restart
isc-dhcp-server stop/waiting
isc-dhcp-server start/running, process 25764

Now, As I switched on c1sus2 from front panel, it booted over network from fb1 like other FE machines and I was able to login to it by first logging to fb1 and then sshing to c1sus2.

Next, I copied the simulink models and the medm screens of c1x06, xc1x07, c1bhd, c1sus2 from the paths mentioned on this wiki page. I also copied the medm screens from chiara(clone):/opt/rtcds/caltech/c1/medm to martian network chiara in the appropriate places. I have placed the file /opt/rtcds/caltech/c1/medm/teststand_sitemap.adl which can be used to open sitemap for c1bhd and c1sus2 IOP and user models.

Then I logged into c1sus2 (via fb1) and did make, install, start procedure:

controls@c1sus2:~ 0$ rtcds make c1x07
buildd: /opt/rtcds/caltech/c1/rtbuild/release
### building c1x07...
Cleaning c1x07...
Done
Parsing the model c1x07...
Done
Building EPICS sequencers...
Done
Building front-end Linux kernel module c1x07...
Done
RCG source code directory:
/opt/rtcds/rtscore/branches/branch-3.4
The following files were used for this build:
/opt/rtcds/userapps/release/cds/c1/models/c1x07.mdl

Successfully compiled c1x07
***********************************************
Compile Warnings, found in c1x07_warnings.log:
***********************************************
***********************************************
controls@c1sus2:~ 0$ rtcds install c1x07
buildd: /opt/rtcds/caltech/c1/rtbuild/release
### installing c1x07...
Installing system=c1x07 site=caltech ifo=C1,c1
Installing /opt/rtcds/caltech/c1/chans/C1X07.txt
Installing /opt/rtcds/caltech/c1/target/c1x07/c1x07epics
Installing /opt/rtcds/caltech/c1/target/c1x07
Installing start and stop scripts
/opt/rtcds/caltech/c1/scripts/killc1x07
/opt/rtcds/caltech/c1/scripts/startc1x07
sudo: unable to resolve host c1sus2
Performing install-daq
Updating testpoint.par config file
/opt/rtcds/caltech/c1/target/gds/param/testpoint.par
/opt/rtcds/rtscore/branches/branch-3.4/src/epics/util/updateTestpointPar.pl -par_file=/opt/rtcds/caltech/c1/target/gds/param/archive/testpoint_211012_174226.par -gds_node=24 -site_letter=C -system=c1x07 -host=c1sus2
Installing GDS node 24 configuration file
/opt/rtcds/caltech/c1/target/gds/param/tpchn_c1x07.par
Installing auto-generated DAQ configuration file
/opt/rtcds/caltech/c1/chans/daq/C1X07.ini
Installing Epics MEDM screens
Running post-build script

safe.snap exists 
controls@c1sus2:~ 0$ rtcds start c1x07
Cannot start/stop model 'c1x07' on host c1sus2.
controls@c1sus2:~ 4$ rtcds list

controls@c1sus2:~ 0$ 

One can see that even after making and installing, the model c1x07 is not listed as available models in rtcds list. Same is the case for c1sus2 as well. So I could not proceed with testing.

Good news is that nothing that I did affect the current CDS functioning. So we can probably do this testing safely from the main CDS setup.

Don't you need to add the new hosts to /diskless/root/etc/rtsystab at fb1? --> There looks many elogs talking about editing "rtsystab".

controls@fb1:/diskless/root/etc 0$ cat rtsystab
#
# host    list of control systems to run, starting with IOP
#
c1iscex  c1x01  c1scx c1asx
c1sus     c1x02  c1sus c1mcs c1rfm c1pem
c1ioo     c1x03  c1ioo c1als c1omc
c1lsc    c1x04  c1lsc c1ass c1oaf c1cal c1dnn c1daf
c1iscey  c1x05 c1scy c1asy
#c1test   c1x10  c1tst2

I have replaced the temporary clamps that were connecting the RC heater to its power supply with a new permanent connection.

In the IY1 rack, I connected the control signal of the RC PID temperature servo - C1:PSL-FSS_TIDALSET - to the input of the RC heater's power supply.

The signal comes from a DAC in the same rack, through a pair of wires connected to the J9-4116*3-P3 cross-connector (FLKM). I joined the pair to the wires of the BNC cable coming from the power supply, by twisting and screwing them into two available clamps of the breakout FKLM in the IY1 rack - the same connected to the ribbon cable from RC Tmeperature box.

Instead of opening the BNC cable coming from the power supply, I thought it was a cleaner and more robust solution to use a BNC-to-crocodile clamp from which I had cut the clamps off.

During the transition process, I connected the power supply BNC input to a a voltage source that I set at the same voltage of the control signal before I disconnected it (~1.145V).

I monitored the temperature signals and it looked like the RC Temperature wasn't significantly affected by the operation.

The Contec test board with Dsub37Fs was on the top shelf of E7

Oct. 15, 2016

Another attempt (following elog 8755) to extract the oven transfer function from time series data using Matlab’s system identification functionalities.

The same time series data from elog 8755 was used in Matlab’s system identification toolbox to try to find a transfer function model of the system.

From elog 8755: H(s) is known from current PID gains: H(s) = 250 + 60/s +25s, and from the approximation G(s)=K/(1+Ts), we can expect the transfer function of the system to have 3 poles and 2 zeros.

I tried fitting a continuous-time and a discrete time transfer function with 3 poles and 2 zeros, as well as using the "quick start" option. Trying to fit a discrete time transfer function model with 3 poles and 2 zeros gave the least inaccurate results, but it’s still really far off (13.4% fit to the data).

Ideas:

1. Obtain more time domain data with some modulation of the input signal (also gives a way to characterize nonlinearities like passive cooling). This can be done with some minor modifications to the existing code on the raspberry pi. This should hopefully lead to a better system ID.

2. Try iterative tuning approach (sample gains above and below current gains?) so that a tune can be obtained without having to characterize the exact behavior of the heater.

Oct. 16, 2016

-Found the raspberry pi but it didn’t have an SD card

-Modified code to run directly on a computer connected to the TC 200. Communication seems to be happening, but a UnicodeDecodeError is thrown saying that the received data can’t be decoded.

-Some troubleshooting: tried utf-8 and utf-16 but neither worked. The raw data coming in is just strings of K’s, [‘s, and ?’s

-Will investigate possible reasons (update to Mac OS or a difference in Python version?), but it might be easier to just find an SD card for the raspberry pi which is known to work. In the meantime, modify code to obtain more time series data with variable input signals.

Summary

In a previous elog, I demonstrated that the RoC mismatch between ETMX and ETMY does not result in appreciable degradation in the mode overlap of the two arm modes. Koji suggested also checking the effect on the contrast defect. I'm attaching the results of this investigation (I've plotted the contrast,   rather than the contrast defect 1-C).

Details and methodology

• I used the same .kat file that I had made for the current configuration of the 40m, except that I set the reflectivities of the PRM and the SRM to 0. 
• Then, I traced the Y arm cavity mode back to the node at which the laser sits in my .kat file to determine what beam coming out of the laser would be 100% matched to the Y arm (code used to do this attached)
• I then set the beam coming out of the laser for the subsequent simulations to the value thus determined using the gauss command in finesse.
• I then varied the RoC of ETMX (I varied the sagittal and tangential RoCs simultaneously) between 50m and 70m. As per the wiki page, the spare ETMs have an RoC between 54 and 55m, while the current ETMs have an RoC of 60.26m and 59.48m for the Y and X arms respectively (I quote the values in the "ATF" column). Simultaneously, at each value of the RoC of ETMX, I swept the microscopic position of the ETMX HR surface through 2pi radians (-180 degrees to 180 degrees) using the phi functionalilty of finesse, while monitoring the power at the AS port of this configuration using a pd in finesse. This guarantees that I sweep through all the resonances. I then calculate the contrast using the above formula. I divided the parameter space into a grid of 50 points for the RoC of ETMX and 1000 points for the microscopic position of ETMX. 
• I fixed the RoC of ETMY as 57.6m in the simulations... Also, the maxtem option in the .kat file is set to 4 (i.e. higher order modes with indices m+n<=4 are accounted for...)

Result:

Attachment #1 shows the result of this scan (as mentioned earlier, I plot the contrast C and not the contrast defect 1-C, sorry for the wrong plot title but it takes ~30mins to run the simulation which is why I didn't want to do it agian). If the RoC of the spare ETMs is about 54m, the loss in contrast is about 0.5%. This is in good agreement with this technical note by Koji - it tells us to expect a contrast defect in the region of 0.5%-1% (depending on what parameter you use as the RoC of ETMY). 

Conclusion:

It doesn't seem that switching out the current ETM with one of the spare ETMs will result in dramatic degradation of the contrast defect...

Misc notes:

1. Regarding the phase command in Finesse - EricQ pointed out that the default value of this is 3, which as per the manual could give unphysical results sometimes. The flags "0" or "2" are guaranteed to yield physical results always according to the manual, so it is best to set this flag appropriately for all future Finesse simulaitons. 
2. I quickly poked around inside the cabinet near the EX table labelled "clean optics" to see if I could locate the spare ETMs. In my (non-exhaustive) search, I could not find it in any of the boxes labelled "2010 upgrade" or something to that effect. I did however find empty boxes for ETMU05 and ETMU07 which are the ETMs currently in the IFO... Does anyone know if I should look elsewhere for these?
   EDIT 17Jun2016: I have located ETMU06 and ETMU08, they are indeed in the cabinet at the X end...
3. I'm attaching a zip file with all the code used to do this simulation. The phase flag has been appropriately set in the (only) .kat file. setLaserQparam.py was used to determine what beam parameter to assign to be perfectly matched to the Y arm. modeMatchCheck_ETM.py was used to generate the contrast as a function of the RoC of ETMX.
4. With regards to the remaining checks to be done - I will post results of my investigations into the HOM scans as a function of the RoC of the ETMs and also the folding mirrors shortly... 

That sounds weird. If the ETMY RoC is 60 m, why would you use 57.6 m in the simulation? According to the phase map web page, it really is 60.2 m.

This was an oversight on my part. I've updated the .kat file to have all the optics have the RoC as per the phase map page. I then re-did the tracing of the Y arm cavity mode to determine the appropriate beam parameters at the laser in the simulation, and repeated the sweep of RoC of ETMX while holding RoC of ETMY fixed at 60.2m. The revised contrast defect plot is attached (this time it is the contrast defect, and not the contrast, but since I was running the simulation again I thought I may as well change the plot). 

As per this plot, if the ETMX RoC is ~54.8m (the closer of the two spares to 60.2m), the contrast defect is 0.9%, again in good agreement with what the note linked in the previous elog tells us to expect...

So, it seems that changing the ETMX for one of the spares will change the contrast defect from ~0.1% to 0.9%. True? Seems like that might be a big deal.

That is what the simulation suggests... I repeated the simulation for a PRFPMI configuration (i.e. no SRM, everything else  as per the most up to date 40m numbers), and the conclusion is roughly the same - the contrast defect degrades from ~0.1% to ~1.4%... So I would say this is significant. I also attempted to see what the contribution of the asymmetry in loss in the arms is, by running over the simulation with the current loss numbers of 230ppm for Yarm and 484ppm for the X arm, split equally between the ITMs and ETMs for both cases, and then again with lossless arms - see attachment #1. While this is a factor, this plot seems to suggest that the RoC mismatch effect dominates the contrast defect...

[Paco, Yuta]

We measured contrast of Michelson fringe in both arms locked and mis-aligned. It was around 90%.
We also measured the contrast of ITM single bounce vs LO beam using BHD DC PDs. It was around 43%.
The measurement depends on the alignment and how to measure the maximum and minimum of the fringe. ITM-LO fringe was also not stable because motions of AS/LO mirrors are large. More tuning necessary.

Background
 - As measured in elog 40m/17012, we see a lot of CARM in AS, which indicates large contrast defect.
 - We want to check mode-matching of LO beam to AS beam.

BHD DC PD conditioning
 - We added DCPD_A and DCPD_B to /opt/rtcds/caltech/c1/scripts/LSC/LSCoffsets3 script, which zeros the offsets when shutters are closed.
 - We also set C1:LSC-DCPD_(A|B)_GAIN = -1 since they are inverted.

Contrast measurement
 - Contrast was measured using channels ['C1:LSC-ASDC_OUT','C1:LSC-POPDC_OUT','C1:LSC-REFLDC_OUT','C1:LSC-DCPD_A_OUT','C1:LSC-DCPD_B_OUT']. For LO, only DCPD_(A|B) are used.
 - We took 15%-percentile (40% for ITM-LO fringe) from the maximum and minimum of the data, and took the median to estimate the maximum value and the minimum value (see Attachment).
 - Contrast = (Imax - Imin) / (Imax + Imin)
 - We measured three times in each configuration to estimate the standard error.
 - Jupyter notebook: https://git.ligo.org/40m/scripts/-/blob/main/CAL/BHD/measureContrast.ipynb

Results
Both arms locked, MICH fringe (15% percentile)
Contrast measured by C1:LSC-ASDC_OUT is 89.75 +/- 0.17 %
Contrast measured by C1:LSC-POPDC_OUT is 79.41 +/- 0.86 %
Contrast measured by C1:LSC-REFLDC_OUT is 97.34 +/- 0.34 %
Contrast measured by C1:LSC-DCPD_A_OUT is 95.41 +/- 1.55 %
Contrast measured by C1:LSC-DCPD_B_OUT is 89.76 +/- 1.49 %
Contrast measured by all is 90.34 +/- 1.68 %

Both arms mis-aligned, MICH fringe (15% percentile)
Contrast measured by C1:LSC-ASDC_OUT is 89.32 +/- 0.57 %
Contrast measured by C1:LSC-POPDC_OUT is 94.55 +/- 0.62 %
Contrast measured by C1:LSC-REFLDC_OUT is 97.95 +/- 1.37 %
Contrast measured by C1:LSC-DCPD_A_OUT is 96.40 +/- 1.04 %
Contrast measured by C1:LSC-DCPD_B_OUT is 90.98 +/- 1.07 %
Contrast measured by all is 93.84 +/- 0.94 %

ITMY-LO fringe (40% percentile)
Contrast measured by C1:LSC-DCPD_A_OUT is 45.51 +/- 0.45 %
Contrast measured by C1:LSC-DCPD_B_OUT is 38.69 +/- 0.43 %
Contrast measured by all is 42.10 +/- 1.03 %

ITMX-LO fringe (40% percentile)
Contrast measured by C1:LSC-DCPD_A_OUT is 46.65 +/- 0.65 %
Contrast measured by C1:LSC-DCPD_B_OUT is 39.82 +/- 0.51 %
Contrast measured by all is 43.24 +/- 1.45 %

Discussion
 - As you can see from the attachment, REFLDC is noisy and over estimating the contrast. ASDC is reliable. We need to tune the threshold to measure the maximum value and minimum value. We should also use the mode instead of median.
 - Contrast depends very much on the alignment. We didn't tweak too much today.
 - ITM-LO fringe was not stable, probably due to too much motion in AS1, AS4, LO1, LO2. Their damping needs to be re-tuned.

Next:
 - Model FPMI sensing matrix with measured contrast defect
 - Estimate AS-LO mode-mismatch using the measured contrast
 - Lock ITM-LO fringe using DCPD_(A|B) as error signal, and ITM or LO1/2 as actuator
 - Lock MICH with DCPD_(A|B), and with LO beam
 - Get better contrast data with better alignment and better AS1, AS4, LO1, LO2 damping

I came to the lab. The control room AC was off -> Now it is on.

Here is the setting of the AC meant for continuous running

 Needs

There were no injuries...Now we need to get some new chairs.

 The control room desk tops heights on the east side were lowered by 127 mm

The control room UPS started making a beeping noise saying batteries need replacement. I hit the "Test" button and the beeping went away. According to the label on it, the batteries were last repalced in March 2016, so maybe it is time for a replacement, @Chub, please look into this.

[Evan, Jenne]

We were sitting trying to lock MICH (hooooorraaaayy!!!), and the emergency lights above the control room door came on, and then ~1 minute later turned off.  Steve, can you see what's up?

I found that the control MEDM screen was left open on the c1vac workstation. This should be closed every time you leave the workstation, to avoid accidental button pressing and such.

The network outage meant that the EPICS data from the pressure gauges wasn't recorded until I reset everything ~noon. So there isn't really a plot of the outgassing/leak rate. But the pressure rose to ~2e-4 torr, over ~4 hours. The pumpdown back to nominal pressure (9e-6 torr) took ~30 minutes.

Begun setting up an environment (as mentioned before, on my local machine) and scripts to run experiments with Convolutional networks for beam tracking. All code has been pushed to this folder in the GigEcamera repository. I am presently looking for pre-processing techniques for the video which go beyond the usual "Crop the images! Normalize pixel values! Convert to Grayscale!".

Worked further on this. I skimmed through a few resources to look for details of what pre-processing can be done. Here (am planning to convert all these resources, particularly those I come across for GANs into either a README on the repo or a Wiki soon) are some of the useful things I found during today's reading. The work I skimmed through today mostly pointed to the use of a median filter for pre-processing, if any is to be done. I am presently using the Sequential() API in Keras to set up the neural network. I will train it tomorrow.

In the previous meeting, Koji pointed out (once again) that I should determine if the displacement values and frames are synchronized before training a network. Pooja did the following last time. Koji also suggested that I first predict the motion (a series of x and y coordintates) and then slide resulting plots around until I get the best match for the original motion. This is however not possible with a neural network based approach as the network learns exactly what you show it and therefore it will learn any mismatch between the labels and the frames and predict exactly that. Therefore I came up with what Koji described as "hacky" method to achieve the same using the opencv work described previously in this elog (the only addition being the application of a mask to block out the OSEMs and work only with the beam spot) .

Hacky technique to sync frames and labels:

1. I ran the OpenCV algorithm on the data to obtain a plot for predicted motion depicted in Attachment #2. As is evident, the predicted motion is only an approximate of the actual motion and also displays a shift . However, a plot of the fourier transform of the signal (see Attachment #1) shows that the components present are the same. However, the predominant frequency component is 0.22 Hz rather than 0.2 Hz as stated by Pooja in her elog. I wonder if this is of any consequence. Therefore, this predicted motion can be slid around until it overlaps with the applied sinusoidal dither signal "well".
2. Defining "well": I computed an error signal as the differnece between the predicted signal and the actual motion with each signal being normalized by subtracting the mean and then dividing the resulting signal by the maximum value (see Attachment #3). The lower the power of the resulting signal, the better the synchronization of the predicted and actual signal. Note: To achieve this overlap of signals, datapoints are removed from either the start or end of the signals and this effectively reduces the number of data points available for training by 36 pionts (see Attachment #4, positive and negative shifts merely indicate if the predicted signal is being moved right or left).
3. Attachments #5,#6 show the resutls of shifting the data by 36 samples. it is evident that there is far greater overlap of the prediction and the actual values.
4. Well, what now? I will use the mapping between labels and frames obtained by the above steps to train a neural network.

[Koji, Milind - 21/06/2019]

1. Well, the above is fine, but why is contour detection really necessary? Why not take a weighted sum of all the pixel values (in a rectangular region obtained, say, after blocking out the OSEMs) to see what the centroid motion is? Black areas (0 pixel intensity values) will not contribute to this sum anyway. Perhaps that can be used for the sliding instead of the above (fallible!) approach, specially for cases in which the beam "spot"  is just a collection of random speckles?
   1. Something like this was done by Pooja where she computed the sum of pixel intensities in a rectangular region containing the beam spot. However, she did this for very noisy data and observed intensity variation at a frequency double that of the applied signal.
   2. Results of applying a median filter and doing the same are presented in Attachment #7. Clearly, they can't be used for this sliding task.
   3. Results of computing the weighted sum of all the coordinates (with pixel intensities as the weights are presented in Attachment #8. Clearly, for this data and for this task, the contour approach seems to be a better method. Further, these resutls just serve to prove Rana's point that such simple, unsophisticated, naive approaches will not produce desired results and therefore, shall be presented in this very context in the report that is due.
2. The contour detection technique does not work if the beam spot is just a cokllection of speckles. In that case Koji suggested that we use a bounding convex hull instead of a contour. Alternately, for a bunch of speckles I can perform dilation to reduce it to the same problem.
3. Using gpstime for time stamping: To determine the absolute time which a frame is grabbed. However, the time between the time being recorded and grabbing of frame needs to be determined for this which should be doable using linux/python commands.

Upcoming work (in the order of priority):

1. Data acquisition: With the mode cleaner being locked and Kruthi having focused on to the beam spot, I will obtain data for training both GANs and the convolutional networks. I really hope that some of the work done above can be extended to the new data. Rana suggested that I automate this by writing a script which I will do after a discussion with Gautam tomorrow.
2. Network training for beam spot tracking: I will begin training the convolutional network with the data pre-processed as described above. I will also simultaneously prepare data acquired from the GigE and train networks on that. Note: I planned to experiment with framewize predictions and hence did some of the work described above. However, I will restrict the number of experiments on that and perform more of those that use 3D convolution. Rana also pointed out that it would be interesting to have the network output uncertainity in the predictions. I am not sure how this can be done, but I will look into it.
3. Simulation:
   1. Putting the physics in: Previously, I worked on adding point scatterers. I shall add the effect of surface roughness and incorporate the BRDF next. Just as Gautam did, Rana also reccommended that I go through Hiro Yamamoto's work to improve my understanding of this.
   2. GANs: I will put together a readme (which I will turn into a wiki later) for all the material that I am using to develop my ideas about GAN training. Currently, my understanding of GANs is that they take as input noise vectors which are fed to the generative networks which then produce the fakes. This clearly isn't the only way to do it as GANs are used for several applications such as image generation from text. I am referring to these papers to set up the necessary architecture.
4. PMC autolocker: I will convert the existing autolocker script to python. Rana also suggested that it would be interesting to see what the best settings of the hyperparameters would be to lock the PMC the fastest. I will write a script to do that and plot a 3D surface plot of the average time taken to lock the PMC as a function of the PZT scan speed and the Servo gain to determine the optimal setting of these "hyperparameters".
5. Cleaning up/ formalizing code: Rana pointed out that any code that messes with channel values must return them to the original settings once the script is finished running. I have overlooked this and will add code to do this to all the files I have created thus far. Further, while most of my code is well documented and frequently pushed to Github, I will make sure to push any code that I might have missed to github.
6. Talk to Jon!: Gautam suggested that I speak to Jon about the machine requirements for setting up a dedicated machine for running the camera server and about connecting the GigE to a monitor now that we have a feed. Koji also suggested that I talk to him about somehow figuring out the hardware to ensure that the GigE clock is the same as the rest of the system.

I discussed this with Gautam and he asked me to come up with a list of signals that I would need for my use and then design the data acquisition task at a high level before proceeding. I'm working on that right now. We came up with a very elementary sketch of what the script will do-

1. Check the MC is locked.
2. Choose an exposure value.
3. Choose a frequency and amplitude value for the applied sinusoidal dither (check warning by Gabriele below).
4. Apply sinusoidal dither to optic.
5. Timestamping: Record gpstime, instantaneous channel values and a frame. These frames can later be put together in a sequence and a network can be trained on this. (NEED TO COME UP WITH SOMETHING CLEVERER THAN THIS!)

Tomorrow I will try and prepare a dummy script for this before the meeting at noon. Gautam asked me to familiarize myself with the awg, cdsutils (I have already used ezca before) to write the script. This will also help me do the following two tasks-

1. IFO test scripts that Rana asked me to work on a while ago
2. The PMC autolocker scripts that Rana asked me work on

I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.

I am pushing the code that I wrote for

1. Kruthi's exposure variation - ccd calibration experiment
2. modified camera_client_movie.py code (currently at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon)
3. interact.py (to interact with the GigE in viewing or recording mode) (currently at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon)

to the GigEcamera repository.

Gautam also asked me to look at Jigyasa's report and elog 13443 to come up with the specs of a machine that would accomodate a dedicated camera server.