Mode cleaner was not locked today. Koji came in and concluded that PSL had died. So we keyed it. Then we ran my unstick.py code. Mode cleaner is locked now.

[koji, gautam]

In addition to c1psl needing a reboot, megatron was un-ssh-able (although it was responding to ping). Clue was that the NPRO PZT control voltage was drifting a lot on the StripTool trace. Koji hard-rebooted the machine. Now IMC is locked, and FSS slow servo is also running.

There were several small/medium earthquakes in Ridgecrest and one medium one in Blackhawk CA at about 2000 UTC (i.e. ~ 2 hours ago), one of which caused BS, ITMY, and ETM watchdogs to trip. I restored the damping just now.

I worked on preparing for the c1iscaux upgrade a bit today.

1. Attachment #1: This shows where the 120 GB solid-state hard-drive and the 2 RAM cards (2GB each) are installed.
   • I found that it required considerable application of force to get the RAM cards into their slots.
   • Note: the 4GB RAM is broken up into two separate physical cards, each 2GB. The labeling is a bit confusing, as each card suggests it is by itself 4GB.
2. OS install for c1iscaux:
   • I followed Jon's instructions (and added some of mine to the wiki page to hopefully make this process even less thinking-intensive).
   • To be able to use the IP address 192.168.113.83, removed "bscteststand" from chiara martian.hosts and rev.113.168.192.in-addr.arpa as the last mention I could find of this machine was from 2009 (and I'm pretty sure it isn't an active unit anymore). I then restarted the bind9 process. 
   • The hostname for this machine is currently "c1iscaux3" for testing purposes, I will change it once we do the actual install.
   • There was an error in the installation instructions to allow incoming ssh connections - it is openssh-server that is required, not openssh-client. This has now been fixed on the wiki page instructions.
3. Acromag static IP assignment:
   • Assigned 2 ADCs (XT1221), 5 DACs (XT1541) and 5 sinking BIO units (XT1111) static IP addresses (and labelled them for easy reference) using the windows laptop and the Acromag IP config utility.
   • I saw no reason not to use the 192.168.114.yyy scheme for the Acromag subnet on this machine, even though c1auxex and c1vac both have subnets with this addressing prefix. For reasons unknown to me, Jon opted to use 192.168.115.yyy for the c1susaux Acromag subnet.
4. Followed the excellent step-by-step to install EPICS, Modbus and Asyn.
   • This took a while, ~1 hour, dominated by the building of EPICS. The other two took only a couple of minutes each.
   • The same combination suggested on Jon's wiki, of Modbus R2-11, EPICS base-7.0.1 and asyn4-33, are the most current at the time of installation.
   • Couple of typos that prevented straight up copy-pasting were fixed on the wiki.
5. Playground for testing new database files:
   • made a directory /cvs/cds/caltech/target/c1iscaux3 and copied over the .db files from /cvs/cds/caltech/target/c1iscaux and /cvs/cds/caltech/target/c1iscaux2 over.
   • Johannes said he did not develop any code to automate the process of translating the old .db files into the new ones for the Acromag - I won't invest the time in developing any either as I think just manually editing the files will be faster. 
   • I think I will follow the c1susaux convention of grouping .db files by the physical electronics system where possible (e.g. REFL11 channels in one file, CM channels in one file etc), as I think this makes for easier debugging.
   • There is an old "PZT_AI.db" file which I think consists completely of obsolete channels.
6. Next steps:
   • Wire up the crate [Chub]
   • Make the database files and modbus files for talking to the Acromags on the internal subnet [Gautam], check the .db files [Koji]
   • Wiring of whitening switching from P1 to P2 connector, Issue #1 in this elog (this will also requrie the installation of the DIN shrouds) [Koji]
   • Soldering of P2 interface boards [Gautam]
   • Bench testing [Gautam, Koji, Chub]
   • Installation and in-situ testing [Gautam, Koji, Chub]

All the required additional parts should be here by the end of the week - I'd like to aim for Wednesday 7/24 for the installation in 1Y3 and in-situ testing. While talking to Rana, I realized that we should also factor in the c1aux slow channels into this acromag crate - there is no need for a separate machine to handle the shutters and illuminators. But let's not worry about that for now, those channels can simply be added later.

[Kruthi, Gautam, Rana]

Gautam installed Atom text editor on Pianosa yesterday.

MC spot position measurement scripts (these can be found in /scripts/ASS/MC directory): 

• Changed the power threshold for MC2 lock loss check from 15000 to 12000 (volts) in the MeasureSpotPositions.py script. This is because, the C1:I00-MC_TRANS_SUM reads a value, usually, greater than 14000 and with 15000 as the threshold, the script will always say the MC isn't locked even though it is!. Also, to account for additional variation we have a margin of 2000.
• Issues with datetime: though MeasureSpotPositions.py was creating a .dat file, MC_spotMeasurement_history.py threw an error because the .dat file's name was not in the required format. I fixed this bug.
• Just running the MeasureSpotPositions.py doesn't enter the results into the log file, instead ./mcassMCdecenter should be run
• MC_spotMeasurement_history.py just plots the spot positions (in mm) vs days since 2013, using the log file. It still has some bugs

I've taken the MC2 analog camera down and put another GigE (unit 151) in its place. This is just temporary and I'll put the analog camera back once I finish the MC2 loss map calibration. I'm using a 25mm focal length camera lens with it and it gives a view of MC2 similar to the analog camera one. But I don't think it is completely focused yet (pictures attached).

...more to follow

gautam - Attachment #3 is my (sad) attempt at finding some point scatterers - Kruthi is going to play around with photUtils to figure out the average size of some point scatterers.

[koji, gautam]

Koji pointed out an important subtlety pertaining to the "LATCH ENABLE" signal line on the CM board. The purpose of this line is to smoothly facilitate the transition of a change in the "multi-bit-binary-outputs", a.k.a. "mbbo", that are controlled by MEDM gain sliders, to the analog electronics on the CM board. Why is this necessary? Imagine changing the gain from 7dB (=0111 in mbbo representation) to 8dB (=1000 in mbbo representation). In order to realize this change, all 4 bits have to change their state. But this almost certainly doesn't happen synchronously, because our EPICS interface isn't synchronous. So at some intermediate times, the mbbo representation could be 0100 (=4dB), or 1111 (=15dB), or many other possible values, which are all significantly different from either the initial value or the desired final state. This is clearly undesirable.

In order to protect against this kind of error, a Latched output part, 74ALS573, is used to buffer the physical digital logic levels from the switches in the analog gain stages. So in the default state, the "LATCH ENABLE" signal line is held "LOW". When a change happens in the EPICS value corresponding to a gain slider, the "LATCH ENABLE" state is quickly toggled to "HIGH", so as to enable the appropriate analog gain stages to be switched, and then again to "LOW", at which point the latch holds its output state. This logic is currently implemented by a piece of code called "latch.o", which is the compiled version of "latch.st", which may be found in /cvs/cds/caltech/target/c1iool0 where it presumably was written for the IMC servo board, but not in /cvs/cds/caltech/target/c1iool0  , which is where the CM board database files reside. The only elog reference I can find pertaining to this particular piece of code is from Alan, and doesn't say anything about the actual logic.

For the new c1iscaux, we need to implement this logic somehow. After discussion between Koji and me, we feel that a piece of python code is sufficient. This would continuously run in the background on the supermicro server machine. The channel hierarchy for each gain channes is as follows (I've taken the example of C1:LSC-CM_REFL1_GAIN):

• C1:LSC-CM_REFL1_GAIN ------ this is the channel tied to an MEDM slider, and so is a "soft" channel
• C1:LSC-CM_REFL1_SET ------- this is a "soft" channel that gets converted to an mbbo
• C1:LSC-CM_REFL1_BITS ------ this is a channel that actually controls (multiple) physical binary outputs on the Acromag

So the logic will be that it continuously scans the EPICS channel C1:LSC-CM_REFL1_GAIN  for a change in set value. When a change is detected, it has to update the C1:LSC-CM_REFL1_SET channel. In the next EPICS refresh cycle, this would result in the mbbo bits, C1:LSC-CM_REFL1_BITS , all changing to the appropriate values. After these changes have happened, we need to toggle the LATCH ENABLE in order to allow the changes to propagate to the analog gain stage switches. Need to think about what's the best way to do this.

I completed the translation of the .db files for the EPICS database records from the VME notation to the Acromag/Modbus/Asyn notation. The channels are now organized into 5 database files, located in /cvs/cds/caltech/target/c1iscaux3/,  for convenience:

1. C1_ISC-AUX_LSCPDs.db -------- This handles whitening gain, AA enable/bypass, Demodulator FE, and PD Interface Board channels for REFL11, REFL55, REFL33, REFL165, POP22, POP110, POX11, POY11, AS55 and AS110 photodiodes.
2. C1_ISC-AUX_CM.db -------------- This handles all channels for the CM board. The mbbo addressing notation needs to be checked.
3. C1_ISC-AUX_QPDs.db ----------- This handles all channels for the IPPOS QPD.
4. C1_ISC-AUX_ALS.db ------------- This handles all channels for the IR ALS DFD LO and RF power monitoring.
5. C1_ISC-AUX_SPARE.db ---------- This handles the unused channels for the various whitening, AA and PD interface boards.

For reasons unknown to me, the database files in the other Acromag system target directories (e.g. c1susaux, c1auxex) all had 755 level access permission - maybe this is required for systemctl to handle the EPICS serving? Anyways, I upgraded the permission level of the above 5 files using chmod.

There are almost certainly typos / other errors, and I may have missed copying over some soft/calibrated channels, but I hope that this way of grouping by subsystem will make the debugging less painful. Once Chub connects up the power lines to the Acromags, I will run the soft tests. For this purpose, I've also made a C1_ISC-AUX.cmd file and a C1_ISC-AUX.env file in the above target directory, and also made the modbusIOC.service file in /etc/systemd/system on the supermicro.

Now that the .db files were prepared, I wanted to test for errors. So I did the following:

1. Acromags were mounted on the DIN rails. Attachment #1 shows the grouping of ADC, DAC and BIO units. They are labelled with their IP addresses.
2. Wiring of power:
   • Chub had already prepared the backplane with the power connectors, switches and indicator LEDs.
   • So I just had to daisy chain the +24 V (RED) and GND (BLACK) terminals for all the acromags together, which I did using 24 AWG wire (we may want to use heavier gauge given the current draw).
3. Ethernet cables were used to daisy chain the network connectivity between the various units.  Attachment #1 shows the current state of the chassis box.
4. Front panel pieces were attached and labelled, see Attachment #2. 
   • I found it was sufficient to use the front - we may use the rear panel slots when we want to add connections for controlling the c1aux machine channels.
   • The D15 P2 connector panel for the CM board will arrive tomorrow and will be installed then.
5. Entire setup was connected to power and ethernet, see Attachment #3. 
   • As usual, the current draw is significant for the collection of Acromags, I got around this problem by using the bench supply to "Parallel" mode to enhance the current driving capacity.
   • For the ethernet connection, I used the office space port #6, which I connected at the network rack end to the eth1 port of the Supermicro.

All the Acromags are seen on the 192.168.114 subnet on c1iscaux3 - however, when I run the modbusIOC process, I see various errors in the logfile , so more debugging is required. Nevertheless, progress.

Update 2245: Turns out the errors were indeed due to a copy/paste error - I had changed the IP addresses for the ADCs from the .115 subnet c1susaux was using, but forgot to do so for the DACs and BIOs. Now, if I turn off the existing c1iscaux so that there aren't any EPICS clashes, the EPICS server initializes correctly. There are still some errors in the log file - these pertain to (i) the mbbo notation, which I have to figure out, and (ii) the fact that this version of EPICS, 7.0.1, does not support channel descriptions longer than 28 characters (we have several that exceed this threshold). I think the latter isn't a serious problem.

Getting closer... Note that I turned off the c1iscaux VME crate to prevent any EPICS server clashes. I will turn it back on tomorrow.

[Kruthi, Yehonathan, Gautam]

Today evening, Yehonathan and I aligned the MC2 cameras. As of now there are 2 GigEs in the MC2 enclosure. For the temporary GigE (which is the analog camera's place), we are using an ethernet cable connection from the Netgear switch in 1x6. The MC2 was misaligned and the autolocker wasn't able to lock the mode cleaner. So, Gautam disabled the autolocker and manually changed the settings; the autolocker was able to take over eventually.

Rana and I talked about some (genius) options for the large range DC bias actuation on the SOS, which do not require us to supply high-voltage to the OSEMs from outside the vacuum.

What we came up with (these are pretty vague ideas at the moment):

1. Some kind of thermal actuation.
2. Some kind of electrical actuation where we supply normal (+/- 10 V) from outside the vacuum, and some mechanism inside the chamber integrates (and hence also low-pass filters) the applied voltage to provide a large DC force without injecting a ton of sensor noise.
3. Use the blue piers as a DC actuator to correct for the pitch imbalance --- Kruthi and Milind are going to do some experiments to investigate this possibility later today.

For the thermal option, I remembered that (exactly a year ago to the day!) when we were doing cavity mode scans, once the heaters were turned on, I needed to apply significant correction to the DC bias voltage to bring the cavity alignment back to normal. The mechanism of this wasn't exactly clear to me - furthermore, we don't have a FLIRcam picture of where the heater radiation patter was centered prior to my re-centering of it on the optic earlier this year, so we don't know what exactly we were heating. Nevertheless, I decided to look at the trend data from that night's work - see Attachment #1. This is a minute trend of some ETMY channels from 0000 UTC on 18 July 2018, for 24 hours. Some remarks:

1. We did multiple trials that night, both with the elliptical reflector and the cylindrical setup that Annalisa and Terra implemented. I think the most relevant part of this data is starting at 1500 UTC (i.e. ~8am PDT, which is around when we closed shop and went home). So that's when the heaters were turned off, and the subsequent drift of PIT/YAW are, I claim, due to whatever thermal transients were at play.
2. Just prior to that time, we were running the heater at close to its maximum rated current - so this relaxation is indicative of the range we can get out of this method of actuation.
3. I had wrongly claimed in my discussion with Rana this morning that the change in alignment was mostly in pitch - in fact, the data suggests the change is almost equal in the two DoFs. Oplev and OSEMs report different changes though, by almost a factor of 2....
4. The timescale of the relaxation is ~20 minutes - what part(s) of the suspension take this timescale to heat up/cool down? Unlikely to be the wire/any metal parts because the thermal conductivity is high? 
5. In the optimistic scenario, let's say we get 100 urad of actuation range - over 40m, this corresponds to a beam spot motion of ~8mm, which isn't a whole lot. Since the mechanism of what is causing this misalignment is unclear, we may end up with significantly less actuation range as well.
6. I will repeat the test (i.e. drive the heater and look for drift in the suspension alignment using OSEMs/Oplev) in the afternoon - now I claim the radation pattern is better centered on the optic so maybe we will have a better understanding of what mechanisms are at play.

Also see this elog by Terra.

Attachment #2 shows the results from today's heating. I did 4 steps, which are obvious in the data - I=0.6A, I=0.76A, I=0.9A, and I=1.05A.

In science, one usually tries to implement some kind of interpretation. so as to translate the natural world into meaning.

[chub, gautam]

Bulb replaced. Projector is back on.

The control room UPS started making a beeping noise saying batteries need replacement. I hit the "Test" button and the beeping went away. According to the label on it, the batteries were last repalced in March 2016, so maybe it is time for a replacement, @Chub, please look into this.

I did a whole lot of hyperparameter tuning for convolutional networks (without 3d convolution). Of the results I obtained, I am attaching the best results below.

Define "best"?

The lower the power of the error signal (difference between the true and predicted X and Y positions), essentially mse, on the test data, the better the performance of the model. Of the trained models I had, I chose the one with the lowest mse.

Attached results:

1. Attachment 1: Training configuration
2. Attachment 2: Predicted motion along the Y direction for the test data
3. Attachment 3: Predicted motion along the Y direction for the training data
4. Attachment 4: Learning curves
5. Attachment 5: Error in test predictions
6. Attachment 6: Video of image histogram plots
7. Attachment 7: Plot of percentage of pixels with intensity over 240 with time

(Note: Attachment 6 and 7 present information regarding a fraction of the data. However, the behaviour remains the same for the rest of the data.)

Observations and analysis:

1. Data:
   1. From attachemtns 2, 3, 5: Maximum deviation from true labels at the peaks of applied dither/motion. Possible reasons:
      1. Stupid Cropping? I checked (by watching the video of cropped frames, i.e visually) to ensure that the entire motion of the beam spot is captured. Therefore, this is not the case.
      2. Intensity variation: The intensity (brightness?) of the beam spot varies (decreases) significantly at the maximum displacement. This, I think, is creating a skewed dataset with very few frames with low intensity pixels. Therefore, I think it makes sense to even this out and get more data points (frames) with similar (lower) pixel intensities. I can think of two ways of doing this:
         1. Collect more data with lower amplitude of sinusoidal dither. I used an amplitude of 80 cts to dither the optic. Perhaps something like 40 is more feasible. This will ensure the dataset isn't too skewed.
         2. Increase exposure time. I used an exposure time of 500us to capture data. Perhaps a higher exposure time will ensure that the image of the beam spot doesn't fade out at the peak of motion.
   2. From attachment 5, Saturated images?: We would like to gun for a maximum deviation of 10% (0.1 in this case) from the true values in the predicted labels (Tbh, I'm not sure why this is a good baseline, I ought to give that some thought. I think the maximum deviation of the OpenCV thing I did at the start might also be a good baseline?). Clearly, we're not meeting that. One possible reason is that the video might be saturated- (too many pixels at 255, bleeding into surrounding pixles) leading to loss of information. I set the exposure time to 500us precisely to avoid this. However, I also created videos of the image histograms of the frames to make sure the frames weren't saturated (Is there some better standard way of doing it?). From attachements 6 and 7, I think it's evident that saturation is not an issue. Consequently, I think increasing the exposure time and collecting data is a good idea.
2. The network:
   1. From attachment 4: Training post 25 epochs seems to produce overfitting, though it doesn't seem too terrible (from attachments 2 and 3). The network is still learning after 75 epochs, so I'll tinker with the learning rate, dropout and maybe put in annealing.
   2. I don't think there is a need to change the architecture yet. The model seems to generalize okay (valdiation error is close to training error), therefore I think it'll be a good idea to increase dropout for the fully connected layers and train for longer/ with a higher learning rate.

P.S. I will also try the 2D convolution followed by the 1D convolution thing now. 

P.P.S. Gabriele suggested that I try average pooling instead of max pooling as this is a regression task. I'll give that a shot.

For some reason, rossa's Xdisplay won't start up anymore. This happened right after the UPS reset. Koji and I tried ~1.5 hours of debugging, got nowhere.

Summary:

The database files for C1ISCAUX seem to work file - the exception being the mbbo channels for the CM board.

Details:

This was just a software test - the actual functionality of the channels will have to be tested once the Acromag crate has been installed in the rack. One change I had to make on the MEDM screen for the LSC PD whitening gains was to get rid of the "NMS" suffix on the EPICS channel names for whitening gain sliders/drop-down-menus. I suspect this has to do with the EPICS version we are using, 7.0.1. Furthermore, AS165 and POP55 no longer exist - I hold off removing them from the MEDM screen for the moment.

Next steps:

From the software point of view, the major steps are:

1. Fix the mbbo channel notation in the database files
2. Write and test the latch enabling code
3. Figure out what scripted tests can be done to test the functionality of the new Acromag box.

I am stopping the EPICS server on the new machine and restarting the old VME crate over the weekend.

I'm not able to get trends of the TM adjustment test that Rana had asked us to perform, from the dataviewer. It's throwing the following error:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 7: connect() failed
datasrv: DataWrite failed: daq_send: Resource temporarily unavailable
T0=19-07-20-01-27-39; Length=600 (s)
No data output.

What channels are you trying to read?

Summary:

SnapPy scripts made to work on Pianosa.

Details:

Of course rossa was the only machine in the lab that could run the python scripts to interface with the GigE camera. And it is totally bricked now. Lame.

So I installed several packages. The key was to install pypylon - if you go to the basler webpage, pypylon1.4.0 does not offer python2.7 support for x86_64 architecture, so I installed pypylon1.3.0. Here are the relevant lines from the changelog:

Camera server is running on a tmux session on pianosa. But it keeps throwing up some gstreamer warnings/errors, and periodically (~every 20 mins) crashes. Kruthi tells me that this behavior was seen on Rossa as well, so whatever the problem is, doesn't seem to be because I missed out on installing some packages on pianosa. Moreover, if the server is in fact running, I am able to take a snapshot - but the camera client does not run.

1. Make the MSE a subplot on the same axes as the time series for easier interpretation.
2. Describe the training dataset - what is the pk-to-pk amplitude of the beam spot motion you are using for training in physical units? What was the frequency of the dither applied? Is this using a zoomed-in view of the spot or a zoomed out one with the OSEMs in it? If the excursion is large, and you are moving the spot by dithering MC2, the WFS servos may not have time to adjust the cavity alignment to the nominal maximum value.
3. What is the minimum detectable motion given the CCD resolution?
4. Please upload a cartoon of the network architecture for easier visualization. What is the algorithm we are using? Is the approach the same as using the bright point scatterers to signal the beam spot motion that Gabriele demonstrated successfully?
5. What is the significance of Attachment #6? I think the x-axis of that plot should also be log-scaled.
6. Is the performance of the network still good if you feed it a time-shuffled test dataset? i.e. you have (pictures,Xcoord,Ycoord) tuples, which don't necessarily have to be given to the network in a time-ordered sequence in order to predict the beam spot position (unless the network is somehow using the past beam position to predict the new beam position).
7. Is the time-sync problem Koji raised limiting this approach?

<Adding details>

See Attachment #2.

Training dataset:

1. Peak to peak amplitue in physical units: ?
2. Dither frequency: 0.2 Hz
3. Video data: zoomed in video of the beam spot obtained from GigE camera 198.162.113.153 at 500us exposure time. Each frame has a resolution of 640 x 480 which I have cropped to 350 x 350. Attachment #1 is one such frame.
4. Yes, therefore I am going to obtain video at lower amplitudes. I think that should help me avoid the problem of not-nominal-maximum value?
5. Other details of the training dataset:
   1. Dataset created from four vides of duration ~ 30, 60, 60, 60 s at 25 FPS.
   2. 4032 training data points
      1. Input (one example/ data point): 10 successive frames stacked to form a 3D volume of shape 350 x 350 x 10
      2. Output (2 dimensional vector): QPD readings (C1:IOO-MC_TRANS_PIT_ERR, C1:IOO-MC_TRANS_YAW_ERR)
   3. Pre-processing: none
   4. Shuffling: Dataset was shuffled before every epoch
   5. No thresholding: Binary images are gonna be of little use if the expectation is that the network will learn to interpret intensity variations of pixels.

Do I need to provide any more details here?

?

see attachment #4.

I wrote what I think is a handy script to observe if the frames are saturated. I thought this might be handy for if/when I collect data with higher exposure times. I assumed there was no saturation in the images because I'd set the exposure value to something low. I thought it'd be useful to just verify that. Attachment #3 has log scale on the x axis.

I'm running the MC2 loss map scripts on pianosa now. The camera server is throwing an error and is not grabbing snapshots :(

Update: I finished taking the readings for MC2 loss map. I couldn't get the snapshots with the script, so I manually took some 4-5 pictures.

Can you please be more specific about what the error is? Is this the usual instability with the camera server code? Or was it something new?

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I wrote (/cvs/cds/caltech/target/c1iscaux3/latch.py) and tested the logic illustrated in Attachment #1. Results of a test are shown in Attachment #2, the various channels change as expected. Note that for negative values of the gain channel, the corresponding "BITS" channel will take on values like 65536 - this is because the mbboDirect data type is a 16 bit data type, and presumably the MSB is the sign bit. A bit mask is applied to this channel before the actual BIO unit bits are set - we should verify that the correct behavior happens, but I don't immediately see any problems.

To me, this is a robust logic, but it will benefit from more sets of eyes giving it a look over. The idea is to run this continuously on the Supermicro machine.

Apart from this, I also fixed some errors in the mbboDirect record syntax - so now I am able to start up the EPICS server without it throwing any error messages. It remains to verify that changing an EPICS gain slider results in the appropriate gain bits being flipped in the correct way (on the hardware side, I think the correct behavior is happening on the software end). For this testing, I turned off the old c1iscaux crate at ~10am, and started up the server on c1iscaux3. I am reverting to the nominal config now (~1pm).

Further testing will require the wiring inside the Acromag chassis to be completed. This should be the priority task for next week.

*Update 1130 22 July 2019: I've now installed the required dependencies on c1iscaux3 and setup the latch.py script to run as a systemctl process dependent on modbusIOC.service.

The camera server keeps throwing the error: failed to grab frames. Milind suggested that it might a problem with the ethernet cable, so I even unplugged it and connected it again; it still had the same issue. One more thing I noticed was, it does take snapshots sometimes with the terminal command caput C1:CAM-ETMX_SNAP 1, but produces a segmentation fault when ezca.Ezca().write(C1:CAM-ETMX_SNAP, 1) ezca.Ezca().write(CAM-ETMX_SNAP, 1) is used via ipython. When the terminal command also fails to take snapshots, I noticed that the SNAP button on the GigE medm screen remains on and doesn't switch back to OFF like it is supposed to.

I think ezca.Ezca().write() takes the string "CAM-ETMX_SNAP" as an argument and not C1:CAM-ETMX_SNAP. See this, line 47. Are you sure this is not the problem?

I found the MC2 face camera disabled(?) and the MC servo board input turned off. I turned the MC locking back on but I don't see any camera image yet.

"bricked" is to mean that it has the functionality of a brick and can be tossed. But rossa seems to have just gotten some software config corruption. I spent a couple hours reinstalling SL7 today as per my previous elog notes and the X display seems to work as before.

i.e. it was fine with the default setup, except for the ole "X chrashes if the mouse goes to left side of screen". As before, I

1. blacklisted the nouvaeu driver (which is used by default)
2. download the NVIDIA driver as per the link
3. run its installation from the no-X terminal

left side of screen is safe again

This time I installed SL7.6 and followed the K Thorne wiki. But its having trouble installing cds-root because it can't find root.

Somebody changed the settings on painosa without elogging anything about it. Why does this keep happening? I thought the point of the elog was to communicate. I think there are sufficient number of problems in the lab without me having to manually reset the control room workstation settings every week. Please make an elog if you change something.

In my script I have used "CAM-ETMX_SNAP" only; while entering it in the elog I made a mistake, my bad!

The MC2 has 2 GigE cameras right now. I'll put back the analog asap.

[Kruthi, Milind]

On Friday, Milind and I performed the pitch adjustment test Rana had asked us to do. Only 1 blue beam in case of ITMX and two in case of ETMY, ETMX and ITMY were accessible. Milind (of mass 72 kg as of 10 May 2019) stood on each of the accessible blue beams of the test mass chambers for one minute and I recorded the corresponding gps time. Before moving to the next beam, we spared more than a minute for relaxation after the standing end time. Following are the recorded gps times. 

PS: For each blue beam relaxation time ~ 1 min after the standing end time

• copied over /etc/fstab lines from pianosa sothat the NFS mounts work correctly
• added symlinks so that the NFS dirs mount in the right dirs
• installed Opera browser
• symlink libsasl2.so.3 -> libsasl2.so.2 and now DTT runs and can get data now and in the past
• DTT can natively produce PDF so you don't have to take screen caps of your camera phone and make a chalk drawing of that anymore
• sitemap/MEDM is working
• after editing fonts.alias as detailed in Thorne wiki, I ran 'sudo xset fp rehash' to get MEDM to notice new fonts. MEDM Scalable fonts are back!!
• installed Grace and now dataviewer works
• ndscope not running: yum install ndscope breaks because it can't find a couple of python34 packages
• tested that AWGGUI also runs and puts in real sine waves

Summary:

My goal tonight was to lock the PSL frequency to be resonant in the XARM cavity, using the PSL+EX beat as the error signal. I was not successful - mainly, I was plagued by huge BR mode coupling in the error signal, and I could not enable the BR notch filter in the control loop without breaking the lock. Need to think about next steps.

Details:

• POX and POY locking was easily restored.
• EX green alignment was tweaked at the end-table. A large YAW correction was required, which I opted to apply on the mechanical mirror mounts rather than the PZTs. GTRX ~0.4 was recovered.
• The arm cavity length was first locked using POX as an error signal 
  • Then I looked at the out-of-loop ALS noise, trusting the DFD's V/Hz calibration (red-trace in Attachment #1).
  • I judged it to be close enough to the benchmark reference (green-trace in Attachment #1), and so decided that I could go ahead and try locking.
• A modified version of the script /opt/rtcds/caltech/c1/scripts/XARM/Lock_ALS_XARM.py was used to transition control from POX to the ALS error signal
  • I found that I had to change the sign of the CARM loop gain for the servo to remain stable (in this config, CARM-->MC2 length, thereby modifying the IMC frequency to keep the PSL resonant in the XARM cavity).
  • I don't know why this sign change was required - we are still sticking to the same convention that the beat frequency increases when the temperature slider for the EX laser is incremented in counts.
  • The script failed multiple times at the BOOST/BR notch filter enabling step.
  • Doing these steps manually, I found that turning the BR notch, FM6, ON destroyed the lock immediately.
  • Motivated by this observation, I looked at the in-loop error signal spectrum, see Attachment #2. Here, the PSL frequency is servoed by the ALS error signal, but the BR notch filter isn't enabled.
  • The Bounce-mode peak is huge - where is this coming from? It is absent in the ALS spectrum when the XARM is locked with POX. So it is somehow connected with actuating on the MC2 suspension? Or is it that the FM6=BounceRoll filter of the XARM loop is squishing the noise when looking at the ALS spectrum in POX lock, i.e. Attachment #1? In which case, why can't I engage FM6 for the CARM loop???

Anyway, now that I have a workable set of settings that gets me close to the ALS lock of the XARM, I expect debugging to proceed faster.

Update 2019 July 23: I looked at the control loop shape today, see Attachment #3. I'm not sure I understand why the "BounceRoll" filter  in this filter bank looks like a resonant gain rather than a notch, as it does for the Oplev or SUSPOS loops for example - don't we want to not actuate at these frequencies because the reason the signal exists is because of the imperfect OSEM/magnet positioning? This does not explain the spectrum shown in Attachment #2 however, as that filter was disabled.

This afternoon Gautam and I assessed what to do about restoring the GigE camera software. Here's what I propose:

• Set up one of the new rackmount Supermicros as a dedicated camera feed server
• All GigE cameras on a local subnet connected to the second network interface (these Supermicros have two)
• Put the SnapPy, pypylon, and pylon5 binaries on the shared network drive. These all have to be built from source.
• All other dependencies can be gotten through the package managers, so create requirements files for yum and pip to automatically install these locally.

I've started resolving the many dependencies of this code on rossa. The idea is to get a working environment on one workstation, then generate requirements files that can be used to set up the rest of the machines. I believe the dependencies have all been installed. However, many of the packages are newer versions than before, and this seems to have broken SnapPy. I'll continue debugging this tomorrow.

Summary:

I succeeded in locking the PSL frequency to the XARM cavity length, with 9 pm RMS (Attachment #1) motion below 1 kHz, by actuating on MC2 to change the IMC length. The locks were pretty stable (~20 minutes) - the dominant cause of lockloss was the infamous ETMX drifting problem.

Details:

1. I did not need to do anything to fix the anomalosly high BR mode coupling I reported yesterday .
   • To test where this could be coming from - I looked at the ALS spectrum again with the XARM length locked to the PSL frequency using POX.
   • Then I compared the spectra with the BR filter in the XARM servo enabled/disabled, see Attachment #2. 
   • There bounce/roll peak heights even with the BR filter disabled is ~x100 smaller than what I reported yesterday (it remained the case today, because without enabling the BR filter in the CARM servo bank, the TRX level was fluctuating wildly at ~16 Hz). 
2. The CARM loop (which is what the PSL frequency was slaved to) had ~150 Hz UGF with ~40 degrees phase margin, see Attachment #3.
3. The quoted RMS sensing noise is if we trust the old POX calibration - may be off by a factor of a few, but probably not an order of magnitude. I'll recalibrate using the free-swinging Michelson technique in the coming days.
4. The two broad humps in Attachment #1, centered at ~180 Hz and ~300 Hz, are present in the XARM lock as well - so it is somehow imprinted on the arm cavity length. Fixing that will improve the RMS noise performance significantly.

My main motivation here is to make some measurements and investigate the SoCal idea using a toy system, i.e. a single arm cavity controlled using ALS, so that's what Craig and I will attempt next.

I have been trying a couple of HDR algorithms, all of them seem to give very different results. I don't know how suitable these algorithms are for our purpose, because they are more concerned with final display. I'm attaching the HDR image I got by modifying Jigyasa's code a bit (this image has been be modified further to make it suitable for displaying). Here, I'm trying compare the plots of images that look similar. The HDR image has a dynamic ratio of 700:1

PS: 300us_image.png file actually looks very similar to HDR image on my laptop (might be an issue with elog editor?). So I'm attaching its .tiff version also to avoid any confusion.

Summary:  I calibrated MC2 pitch and yaw offsets to spot position in mm. Here's what I did:

1. Changed the MC2 pitch and yaw offset values using  ezca.Ezca().write('IOO-MC2_TRANS_PIT_OFFSET', <pitch offset value> ) and ezca.Ezca().write('IOO-MC2_TRANS_YAW_OFFSET', <yaw offset value> )
2. Waited for ~ 700-800 sec for system to adjust to the assigned values
3. Took snapshots with the 2 GigEs I had installed - zoomed in and zoomed out. (I'll be using these to make a scatter loss map, verify the calibration results, etc)
4. Ran the mcassDecenter script, which can be found in /scripts/ASS/MC. This enters the spot position in mm in the specified text file.

Results:  In the pitch/yaw vs pitch_offset/yaw_offset graph attached,

• intercept_pitch = 6.63 (in mm) ,  slope_pitch = -0.6055 (mm/counts) 
• intercept_yaw = -4.12 (in mm) ,  slope_yaw = 4.958 (mm/counts) 

Moved the unstick.py code to the ifotest repository here. It now handles signals like those generated by Ctrl-C and so forth. It can still be run as python unstick.py <machine1> <machine2> etc.

I upgraded Pylon, the C/C++ API for the GigE cameras, to the latest release, 5.2.0. It is installed in the same location as before, /opt/rtcds/caltech/c1/scripts/GigE/pylon5, so environment variables do not change. The old version, 5.0.12, still exists at opt/rtcds/caltech/c1/scripts/GigE/backup_pylon5.

The package contains a GUI application (/bin/PylonViewerApp) for streaming video. The old version supports saving still images, but Milind discovered that the new version supports saving video as well. This required installing a supplementary package supporting MPEG-4 output.

Basler's GUI application is launched from the terminal using the alias pylon. I've tested it and confirm it can save both videos and still-image formats. I recommend to also try grabbing images using this program and check the bit resolution. It would be a useful diagnostic to know whether it's a bug in Joe B.'s code or something deeper in the camera settings.

At the lab meeting today, Rana suggested that I use the Pylon app to collect more data if that's what I need. Following this, Jon helped me out by updating the pylon version and installing additional software to record video. Now I am collecting data at

1. higher exposure rate - 600 us magically gives me a saturation percentage of around 1%, see attachment #1 (i.e around 1% of the pixels in the region containing the beam spot are over 240 in value). Ths is a consequence of my discussion with Gabriele where we concluded that I was losing information due the low exposure rate I was using.
2. For much longer: roughly 10 minutes
   1. at an amplitude of 40 cts for 0.2 Hz
   2. at an amplitude of 20 cts for 0.2 Hz
   3. at an amplitude of 10 cts for 0.2 Hz
   4. at an amplitude of 40 cts for 0.4 Hz
   5. at an amplitude of 20 cts for 0.2 Hz
   6. Random motion

Consequently I have dithered the MC2 optic from around 9:00 PM.

Since there are multiple SURF projects that rely on the cameras:

1. I moved the new installs Jon made to "new_pylon5" and "new_pypylon". The old installs were moved back to be the default directories.
2. The bashrc alias for pylon was updated to allow the recording of videos (i.e. it calls the PylonViewerApp from new_pypylon).
3. There is a script that can grab images at multiple exposures and save 12-bit data as uint16 numpy arrays to an HDF5 file. Right now, it is located at /users/kruthi/scripts/grabHDR.py. We can move this to a better place later, and also improve the script for auto adjusting the exposure time to avoid saturations.

My changes were necessary because the grabHDR.py script was throwing python exceptions, whereas it was running just fine before Jon's changes. We can move the "new_*" dirs to the default once the SURFs are gone.

Let's freeze the camera software config in this state until next week.

Somehow I never got around to doing the pixel sum thing for the new real data from the GigE. Since I have to do it for the presentation, I'm putting up the results here anyway. I've normalized this and computed the SNR with the true readings.

SNR = (power in true readings)/ (power in error signal between true and predicted values)

Attachment #2 is SNR of best performing CNN for comparison.

I'll keep developing the camera server on a parallel track using the "new_..." directory naming convention. One thing I forgot to note is that the new pylon/pypylon packages require Python 3, so will not work with any of the 2.7 scripts. All of the environment I need to set up is exclusively Python 3. I won't change anything in the Python 2.7 environment in current use.

Also, I found the source of the bit resolution issue: Joe B's code loads a set of initialization parameters from a config file. One of them is "Frame Type = Mono8" which sets the dynamic range of the stream. I'll look into how this should be changed. 

Summary:

1. There are some broad peaks in the ALS out-of-loop noise, centered at ~145 Hz, ~245 Hz and ~570 Hz which are absent in both the POX in-lock error signal and in the green PDH error signals (see Attachment #1). So I conclude they originate in the IR ALS beat chain somewhere. Needs more investigation, in the general quest to improve the ALS noise.
2. This measurement also shows that the ALS noise is limited by unsuppressed EX green PDH frequency noise above ~400 Hz (100 Hz if you ignore the unexplained broad humps).

These spectra were taken with the arm cavity length locked to the PSL frequency using POX as an error signal, and the EX laser frequency locked to the XARM cavity length by the analog PDH servo at EX, so there is no feedback control with the ALS beat signal as an error signal.

Other details:

• The transition of arm resonance control from POX to ALS error signal is more robust now - I am able to do this during daytime, and also maintain the lock for >20 minutes at a time.
• Rana encouraged me not to spend too much time on this - so my next goal here will be to get the Y arm IR ALS working, and then we can control the two arms using ALS error signals in the CARM/DARM basis instead of the X/Y basis.
• I still think it's worth getting the ALS good enough that the locking becomes repeatable and reliable.
  • The main task here is going to be re-doing the EY green layout to match the EX layout, get good MM into the cavity etc.
  • The IR light also has to be coupled into the fiber at EY.

I brought one CPU (Dell T3500) and one 28" monitor from Mike Pedraza's office in Downs to the 40m. It is on Steve's desk right now, pending setup. The machine already has Solidworks and Altium installed on it, so we can set it up at our leisure. The login credentials are pasted on the CPU with a post-it should anyone wish to set it up.

[yehonathan, gautam]

• we placed a PDA520 photodiode in the AS beampath, so AS110 and AS55 no longer see any light.
• ITMX and ETMX were misaligned (since the plan is to measure the Y arm loss).
• The PDA520 and MC2 transmission are currently going to the Y arm ALS beat channels in the DAQ system. Unfortunately, we have no control over the whitening gains for these channels because of the c1iscaux2 situation.

1. X arm is totally misaligned in order to measure the Y arm loss using the reflection method. Each measurement round consists of measuring the reflected power when the Y arm is aligned and when it is misaligned.

2. The measurement script used is /scripts/lossmap_scripts/armLoss/measureArmLoss.py. It generates a log file in the /logs folder specifying the alignment and misalignment times.

3. The data extraction script dlData.py processes the raw data in the log file and creates a hdf5 file in the /Data folder conataining the data of the measurement it self.

4. dlData.py labels the the aligned and misaligned datas incorrectly when the number of measurement is odd. I use only even number of measurements then.

5. In order to clip the chaotic transition between the aligned and misaligned states I use tDur attribute smaller than the actual sleep time used in the measurement script itself.

6. plotData.py (written by Gautam) and AnalyzeLossData.ipynb (written by me) can be used to calculate the loss and to plot some analyses (see https://nodus.ligo.caltech.edu:8081/40m/14568). They give roughly the same answer. The descripancy can be explained by the different modulation and ITM transmissions used.

7. I take a measurement of 8 repeatitions. I plot the measured reflected power alternating between the aligned and misaligned states. 

I find that the reflected power is repeatable to within 1%.

This is consistent with the transmission data plotted here which is also repeatable to within 1%.

8. I take an overnight measurement of 100 repeatitions.

1. IMC would not lock. c1psl EPICS channels were unresponsive. I keyed the crate and went through the usual burtrestore/PMC-relocking dance.
2. While at 1X2, I decided to take this opportunity to clean up the AG4395 setup that has been setup there unused for several weeks now.
   • Unplugged the active probe connected via BNC-T connector to the mixer IF output.
   • Noticed that the active probe (S/N 2850J01450) did not have it's power connection connected. According to the manual, this is bad. I don't know if the probe is damaged or not.
   • Moved the AG4395 cart out of the way so that there is a little more room around 1X1/1X2.

Summary:

We need to determine the geometry (= round-trip length and RoC of curved mirrors) of the OMC cavities for the 40m BHD experiment. Sticking to the aLIGO design of a 4 mirror bowite cavity with 2 flat mirrors and 2 curved mirrors, with a ~4deg angle of incidence, we need to modify the parameters for the 40m slightly on account of our different modulation frequencies. I've setup some infrastructure to do this analytically - even if we end up doing this with Finesse, it is useful to have an analytic calculation to validate against (also not sure if Finesse can calculate HOMs up to order 20 in a reasonable time, I've only seen maxtem 8). 

Attachment #1: Heatmap of the OMC transmission for the following fields:

• Carrier TEM00 is excluded, but HOMs up to m+n=20 included for both the horizontal and vertical modes of the cavity.
• f1 and f2 upper and lower sidebands, up to m+n=20 HOMs for both the horizontal and vertical modes of the cavity, including TEM00.
• Power law decay assumed for the HoM content incident on the OMC - this will need to be refined
• The white region is where the cavity isn't geometrically stable.
• Green dashed line indicates a possible operating point, white dashed line indicates the aLIGO OMC operating point. On the basis of this modeling, we would benefit from choosing a better operating point than the aLIGO OMC geometric parameters.

Algorithm:

1. Compute the round-trip Gouy phase, , for the cavity.
2. With the carrier TEM00 mode resonant, compute the round-trip propagation phase, , and the round-trip Gouy phase,  for the  mode of the field, with  specifying the offset from the carrier frequency (positive for the upper sideband, negative for the lower sideband). For the aLIGO cavity geometry, the 40m modulation sidebands acquire ~20% more propagation phase than the aLIGO modulation sidebands.
3. Compute the OMC transmission for this round-trip phase (propagation + Gouy).
4. Multiply the incident mode power (depending on the power law model assumed) by the cavity transmission.
5. Sum all the fields.

Next steps:

1. Refine the incident mode content (and power) assumption. Right now, I have not accounted for the fact that the f2 sideband is resonant inside the SRC while the f1 sideband is not. Can we somehow measure this for the 40m? I don't see an easy way as it's probably power dependent?
2. Make plots for the projection along the slices indicated by the dashed lines - which HOMs are close to resonating? Might give us some insight.
3. What is the requriement on transmitted power w.r.t. shot noise? i.e. the colorbar needs to be translated to dBVac.
4. If we were being really fancy, we could simultaneously also optimize for the cavity finesse and angle of incidence as well.
5. Question for Koji: how is the aLIGO OMC angle of incidence of ~4 degrees chosen? Presumably we want it to be as small as possible to minimize astigmatism, and also, we want the geometric layout on the OMC breadboard to be easy to work with, but was there a quantitative metric? Koji points out that the backscatter is also expected to get worse with smaller angles of incidence.

The code used for the ABCD matrix calcs have been uploaded to the BHD modeling GIT (but not the one for making this plot, yet, I need to clean it up a bit). Some design considerations have also been added to our laundry list on the 40m wiki.