[Gautam, Kruthi]

This afternoon, Gautam helped me setup Giada to access the GigE installed for MC2. Unlike Paola, which was being used earlier, Giada has a better battery life and doesn't shutdown when the charger is unplugged. Gautam configured Giada to enable its wireless connection to Martian, just like Koji had configured Paola (https://nodus.ligo.caltech.edu:8081/40m/14672). We also rerouted  the ethernet cable we were using with the PoE adaptor from Netgear Switch in 1x2 to 1x6.

Attachment #1 - In loop error spectra, measured as Koji posted end of last week.

• Main difference is that the line noise seems much lower.
• For the "dark" measurement, I set the IN1 gain of the servo board to the value of +4 dB, which is what it is in lock.
• As Koji mentioned, this isn't an apple-to-apple comparison as the IMC loop will squish the plotted orange trace.
• Nevertheless, the fact that the blue trace is above orange everywhere gives confidence that we are in fact measuring frequency noise.
• For the higher frequency measurement, I used the AG4395 analyzer, which has 50 ohm input impedance. So to get the measurements with the SR785 to line up, I multiplied these by x2.
• For the frequency axis calibration, I used the value of 13 kHz/V for the PDH discriminant, which was what I measured it to be last year (but I didn't check again today).
• Note that the IMC locking loop OLTF has not been undone, so this isn't the actual laser frequency noise on the transmitted beam. In order to measure the latter, we'd have to use (for example) an arm cavity as an analyzer.

Attachment #2 - OLTF of the IMC loop.

• Measurement was made using the IN1/IN2 method, injection was done at the "A EXC" front panel BNC input.
• For comparison, I've overlaid a measurement from the 2017 IMC loop investigations. Doesn't seem to be significantly different.
• UGF and phase margin are in the ballpark of what they were reported to be in the past.

Attachment #3 - Photo courtesy Koji showing the bank of BNC connectors used for these measurements.

Clearly, these measurements were taken in a time when the IMC was "well behaved". How to characterize what's happening when this isn't the case?

Over the last 24 hours, the IMC autolocker was able to keep the MC locked ~60% of the time. This is not particularly good, but is an improvement on ~2 weeks ago when the IMC couldn't be locked.

There are two periods, which I've indicated by vertical cursors, between which the autolocker was doing something strange - usually this kind of trend is caused by one or more of the VME crates being unresponsive and the autolocker gets stuck, but I confirmed that both c1psl and c1iool0 are telnet-able. So I conclude that the stability and reliability of the IMC loop is still not as good as it used to be.

Note also that while the PC drive RMS level mostly hovers around 1 V, there are several excursions above that level. This in itself isn't a new phenomenon. I will do some more characterization by measuring the in-loop error signal spectrum and maybe the OLTF of the IMC locking loop.

Note that I have removed an SR785, an oscilloscope, some SRS instruments from the PSL and PMC last night.

But they (and RF Network Analyzer) were not there when the problem started.

We should record the IMC error (at test point monitor) too? If the IMC locks on Monday too, I'll do it.

After typing up the elog, I decided to try locking the IMC again - now it locks again with the "OLD" gain settings. I tested it ~5 times, the autolocker brings the lock back and the PC drive levels are normal. IMC transmission and MC REFL DC light levels in lock are normal. The PC Drive RMS voltage is <1V. What's more, there is no longer any evidence of 60 Hz line harmonics any more in the PMC diagnostics channels. Compare attachment #1 to this elog.

WTF.

I undid the changes Koji made to the autolocker gains, and am trying the old settings again. Let' see how stable or otherwise the config is. I must've jiggled some poor cable connection back into a good spot while working on the PSL table?

Anyway, this helps Kruthi and Milind.

Summary:

Over the last few days, I've been doing some (complementary) measurements to what Aaron and Koji have been looking at. The motivation was to identify if the problems we are seeing are optical (i.e. imprinted on the PSL light) or electronic. My findings:

1. 60 Hz line noise in PMC REFL and PMC TRANS is heavily dependent on whether I connect cables between the measuring PDs and Acromag ADC or not - but even with the Acromag cable disconnected, the 60 Hz RIN is HUGE - 10 mVpp out of 670 mV DC, and lines are much dirtier if you have connections to the SLOW ADCs. Measurement was made by looking at the time-domain signals on a battery powered Tektronix oscilloscope. See Attachment #1. I believe this line noise is higher it was. Cause is unknown to me at this point.
2. The NPRO noise eater seems to function as advertised. The measured RIN with the noise eater enabled (our nominal operating condition) is in line with what the manual tells us it should be. See Attachment #2.
3. There isn't strong evidence of excess frequency noise (measured with PLL) out to 100 kHz. I didn't measure the high-frequency part yet, but maybe I'm doing something wrong with the PLL setup which should be first corrected. See Attachments #3, #4.
4. The beat note frequency between the free-running PSL and EX NPRO's is definitely slewing more than the quadrature sum of the advertised 1 MHz/min slewing per the manual.

Evidence:

Attachment #1: Time domain look at PMC Refl and Trans signals under various operating conditions. During this work, I took the chance to remove ~4 BNC T connectors that were connected on the PMC TRANS photodiode (Thorlabs). Now, there is one cable going to the Acromag ADC, and one going to the Oscilloscope used to monitor these signals. Any further T-ing can be done at the oscilloscope.

Attachment #2: RIN measurement of the NPRO light. I opted to place a Thorlabs PDA55 in the IR ALS pickoff light path. This is before the light sees the PMC. A DC block was inserted between the PDA55 and the AG4395 used to make the measurement. DC level of the PD output was 3.1 V into high-Z and I used half this value to normalize the measurement made by the 50-ohm input AG4395 into RIN units. The measurement was made with the PZT and slow temperature controls to the NPRO connected/disconnected, but I saw no significant difference. 

Attachment #3: Frequency noise measurement via PLL. This shows the loop transfer funtion for the PLL. Some details of the setup:

• The beat note for locking the PLL was made between the PSL NPRO and the EX NPRO (output of the IR ALS BeatMouth). ~4dBm beatnote.
• Local oscillator was sourced by a Marconi, f_carrier=33 MHz, RF level = +10dBm.
• Level 7 Mixer and LB1005 controller from the mode-spectroscopy PLL setup.
• PLL control signal routed to EX NPRO PZT via Heliax cable running along south arm. 
• Why EX and not PSL or Marconi FM? Latter has limited range, ~1/10th of that offered by NPRO PZT. PSL PZT has a 2.9 Hz corner freq Pomona box. I could disconnect this for the purpose of PLL locking, but I thought it may be interesting to see if there’s any hints of the problem being electrical, by looking at PLL spectra with / without Pomona box. The expected delay due to cabling is only 400 ns, so not really a limiting factor for the PLL bandwidth.
• LB 1005 settings:
  • PI corner = 3 kHz.
  • G = 2.30 (I could not increase this further - with the PSL+Lightwave NPRO PLL, we could achieve a UGF of ~60 kHz, but in this setup, I can't do much better than ~7kHz before the loop starts oscillating, not sure if the fact that the PZT actuation coefficient for the Innolight is ~5x lower than for the Lightwave is enough to explain this?).
  • LFGL = 90 dB.
• Mixer output had a maximum value of 800 mVpp => PLL discriminant is 400 mV/rad.
• The "eye fit" is just the transfer function of two poles at DC (one for frequency to phase conversion in the PLL and one for the LB1005 integrator), and a zero at 3kHz (PI corner). I scaled the gain till the "fit" and measurement lined up, and then used this model to undo the loop suppression of the error signal to extract the frequency noise without worrying about the frequency vector of the measurement being limited.
• Once again, slow temperature control and PZT controls to the PSL NPRO were disconnected so this measurement was made with two free-running NPROs.

Attachment #4: Frequency noise measurement via PLL. This shows the frequency noise. I've overlaid the expected frequency noise between 2 free-running NPROs, model used is in the text box in the plot. There isn't strong evidence of excess high frequency noise in this measurement. The fact that the "LB 1005 input terminated" trace is below all the others supports the hypothesis that I'm measuring real frequency noise. The bump around a few kHz could indicate some gain peaking?

However, I'm unable to find good agreement between the measured frequency noise using the error point and the control point. For the former, I used the PLL discriminant mentioned above of 400 mV/rad, and undid the loop suppression, and for the latter I used a PZT discriminant of 1.7 MHz/V. However, there is still a constant scale difference between these two traces. So I'm doing something wrong?

Next steps:

1. More interpretation of the PLL measurement results required.
2. Measure the PLL error signal spectrum to higher frequencies using the AG4395. 
3. ???

I've not disturbed the PLL setup in case anyone else wants to repeat these measurements, but I have restored the normal electrical connections to the PSL PZT and temperature control.

Some other activity:

1. Alignment into the PMC was tweaked.
2. NPRO laser pump current was increased from 1.9 A to 2.0 A.
3. PMC servo gain was changed from +18 to +17 to prevent the servo from oscillating.

The IMC REFL error signal was measured to compare it with the other spectra (if we have).

The blue curve is the in-loop IMC error and the red is the dark noise. So they are not an apple-to-apple comparison. But the red noise is going to be suppressed by the loop, and still the red is below blue. This means that the blue curve is the measured noise rather than the readout noise.

We suspect that the current issue is the PC drive saturation (as usual). Does this indicate that the laser freq noise is actually increased?

----

Another suspect was that the degradation of the LO level. We used to have the issue of slowly dying ERA-5 (ERA-5SM indeed). The RF levels on the demod board were measured using an active probe.

The LO input: 0dBm, ERA-5 input: -2.7dBm and -2.1dBm for I and Q. I found that the outputs of the ERA-5SM were +10.5dBm and +10.6dBm.
This lead me to replace the chips but the situation was not changed. Then I realized that the LO levels should have been measured with the load replaced from the mixers to a 50Ohm load. Somehow these mixers lower the apparent LO levels. So I decided to say this is OK.

I think a Faraday rotator rotates the polarizations in a same way for both forward and backward beam, and it's not like in this figure.
And the transmission through multiple faradays will also be a big issue.

Here are the results from the fit. Data can be found on nodus in /users/aaron/40m/data/PMC/190617/. I've put a jupyter notebook with the analysis in /users/aaron/40m/analysis/PDH_calibrate.ipynb (might be some filename issues due to different directory structure on my laptop).

Here's a summary of the current measurement. I'll be referencing the diagram for the PMC servo card.

1. With the PMC servo loop open, sweep the PMC PZT by sending a triangle wave in to J5 (external drive on the servo card).
   1. I used a 100Hz drive, but should use something slower so my drive isn't filtered out by the 100Hz pole on the servo card and the 10Hz pole on the PZT.
2. Monitor the voltage at the HV drive, as well as at "mixer out" (J8 on the servo board)
   1. Note that I took this PDH error signal from FP2TEST rather than "mixer out", which means my error signal was not low pass filtered.
3. Calibrate the HV mon in units of Hz by fitting the PDH signal. The sidebands should be 35.5Mhz away from the carrier peak.
   1. This part needs to be done differently to account for thermal locking in the PMC.
4. You now have the PDH error signal as a function of PMC resonance in Hz, and can use that to calibrate the PDH error spectrum.
   1. The spectrum is taken when the PMC is locked, so the Hz/V scaling is the slope of the PDH error signal.

In the figures below, I obesrved that for fast (100Hz) drives, the PDH error signal had a pi/2 phase shift relative to the triangle wave, which means even though the resonance appears near the turnaround of the triangle, it is actually occuring near the center of the range.

There are several problems with this data:

• PMC error signal spectrum is not properly calibrated, even according to the process described above
• The drive was faster than the response of the PZT.
• I was driving with a ~1V excitation, so I've lost a factor of 10 somewhere on the way to the external drive curve. Probably just a problem with how I've read the data dump from the scope.

Worked further on this. I skimmed through a few resources to look for details of what pre-processing can be done. Here (am planning to convert all these resources, particularly those I come across for GANs into either a README on the repo or a Wiki soon) are some of the useful things I found during today's reading. The work I skimmed through today mostly pointed to the use of a median filter for pre-processing, if any is to be done. I am presently using the Sequential() API in Keras to set up the neural network. I will train it tomorrow.

I made a script (/users/aaron/40m/GPIB/tds_dump.py) that grabs data from a Tektronix scope and packages into a pickled dict with the following structure:

1. ch1
   1. times ("ts")
   2. values ("vals")
   3. channel info ("info")
2. ch2
   1. ""
3. etc

I made a python notebook that does the following:

1. Grab the data from the pickle above
2. Fit a triangle wave to the drive signal
3. Determine the (change in Volts) / second from the triangle wave, as well as define the times of a single sweep of the PDH error signal
4. Trim the error signal data to contain the PDH signal from the carrier and two sidebands only (the original trace was for three periods).
5. Fit the functional form of the PDH signal to the trimmed error signal. 
   1. The sideband frequency is fixed at 35.5 MHz, and the scaling of Volts-to-Hz is left free, so this fit gives the calibration of IF volts to Hz.
6. Grab the spectra (already saved from the Agilent with the netgpib scripts) and apply this V-to-Hz scaling
7. Plot the spectra

The fit in step (5) is still looking quite bad, despite the fitted values being close to the expected. Since we really just want a calibrated spectrum, I'll instead fit a line to the linear portion of the PDH error signal for the carrier and both sidebands, then determine the scaling from that.

As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation. 

Milind pointed out that all boxes on the medm screens were white. I didn't have diagnostics from the medm screens, so I started following the troubleshooting steps on the restart procedures page.

It seemed like maybe a frontend problem. I tried telnet-ing into several of the fe, and wasn't able to access c1psl. The section on c1psl mentions that if this machine crashes, the screens will go white and the crate needs to be turned off and on. Millind did this.

Now, most of the status lights are restored (screenshot).

Milind: I did a burtrestore following this and locked the PMC following the steps described in this elog.

Begun setting up an environment (as mentioned before, on my local machine) and scripts to run experiments with Convolutional networks for beam tracking. All code has been pushed to this folder in the GigEcamera repository. I am presently looking for pre-processing techniques for the video which go beyond the usual "Crop the images! Normalize pixel values! Convert to Grayscale!".

Grabbed the PMC data

I went to set up the spectrum analyzer measurements through GPIB, but inadvertently deleted the contents of ~/Agilent/netgpibdata/ (made a soft link in my folder, decided I wanted it gone but rm'ed instead of unlink). I copied what I think was in that folder back (from /opt/rtcds/caltech/c1/scripts/general/labutils/netgpibdata).

Again, the spectra are:

• R-- PMC TRANS PD into SR560 with G=2 DC coupled, no filtering
• A-- PDH IF into SR560 with G=2, DC coupled, no filtering
• B-- PMC PZT HV MON into SR560 with G=2, AC coupled, no filtering

I recorded the three spectra with the following parameters:

• Three separate spans:
  • 10 Hz to 150 kHz
  • 100 to 550 kHz
  • 500 kHz to 2.5 MHz
• bwSpanRatio = 0.1 %
• averages: 10
• number of points: 801
• spec type: noise (PSD units)

I then ran AGmeasure with the above parameters in the yaml, with the rest following the defaults in AgilentTemplate.yaml. I saved the data in /users/aaron/40m/data/PMC/190617/

Looks like the header contains all of the parameters, so I shouldn't have trouble distinguishing the spectra. I didn't get the instant plotting working, but the data seem to be there.

I'm still having trouble getting the data from the oscilloscope. I'm not sure why the tektronix scrips I've used before aren't working, I'm checking it now.

update: Grabbed the data, the issue was just using the wrong IP address. 

The analog camera is aligned and we are able to see all the 4 OSEMs (pictures attached). Due to secondary reflection from the beamspiltter (BS1-1064-33-2037-45S), when the MC2 is locked, we are getting a ghost image of the beam spot along with the primary image. 

The pylon app in Paola was reporting an error saying "0xE1000014: The buffer was incompletely grabbed". I followed the instructions given in this site, and changed the 'Packet Size' to 1500 and 'Inter-Packet Delay parameter' to a value greater than 20,000 (µs). This did the trick and I was able to use the continuous shot mode without any interruption. I'm attaching a picture of MC2 that I captured using GigE.

The circuit diagram for the PMC servo card is D980352. From this diagram, I see that I can send an excitation from the network analyzer to FP2TEST (9.09 kOhm input impedance) where it is added to the PMC error signal before going to the loop filters.

I hook up the following

• Agilent 4395A output to SR560 (300 Hz HP, gain of 1)
• SR560 to FP2TEST and to Agilent's channel R
• PMC error signal IF (mixer box mounted to rack, I noticed the IF BNC->SMA were a bit loose/stressed by a hanging LP RF filter) to SR560 (also 300 Hz HP, gain of 2)
• SR560 to Agilent channel A

I 'Enable' Test 2 on the PSL screen, so FP2TEST gets added to the error signal.

PDH signal

• TDS 3034B with four channels
  • 1. PMC servo box external drive (split off from the function generator)
  • 2. PMC servo box output monitor (mirrors the drive, shows when drive is saturating)
  • 3. IF signal (split off after the mixer)
  • 4. PMC Trans (long BNC from the PSL table)
• SRS DS345 function generator (into the PMC servo box' external drive)
  • 100 Hz signal (there's a 10 Hz pole on the PZT drive, so any faster than this and I can't see both sidebands without the HV output mon clipping)
  • 3.19 Vpp amplitude (smallest amplitude at 100 Hz such that both sidebands are well resolved)
  • 1.52 V offset (center the carrier's PDH error signal at pi/2 out of phase with the drive)

I was able to see the carrier and both sidebands.

I tried to grab this data from the scope via ethernet, but was unsuccessful (timeout errors, I'm using the scripts from scripts/tektronix/tek-dump, and the GPIB box that Kruthi had been using for the GigE cam; I also tried plugging in directly scope->ethernet. Never got anything but timeout errors, so maybe I'm not specifying the port correctly. Anyway the trace is frozen on the scope for later use, or I can easily repeat this now that I know how).

Spectrum

Next, I locked the PMC (Test1 is off, tune DC output adjust until I get some transmission, turn on the loop at Test1, increase the gain to before the loop goes unstable). I'm sending the following channels to SR560 (gain = 2, no filtering, high dynamic reserve, 50 Ohm outputs), and reading spectra from the Agilent 4395A:

• R-- PMC TRANS PD
• A-- PDH IF
• B-- PMC PZT HV MON

The HV mon was always saturating the preamp, so I disconnected it; I added a 50 Hz (6db) high pass to the Trans PD signal, since it has a DC component.

I got to take a look at the traces on the spectrum analyzer front panel, but too tired to do the GPIB for now. There are peaks, things look reasonable.

Today, I tried aligning it further; I'm attaching a picture of it. We are not able to see all the 4 OSEMs yet. In the reference picture I had taken, before taking off the previous analog setup, the OSEMs are not seen. So, I don't really understand what the other 2 spots seen on the current screen are. Are they actually OSEMs?

I need a laptop next to MC2, so that I can have a look at it and make further alignments. So, I tried accessing the GigE attached to the telescope using Paola. The pylon app in it, throws an error, few seconds after running it in continuous shot mode, and disconnects the GigE; everything works fine on Rossa though. I'll put up further details soon.

SURFS want some locking of IMC for camera adjustment.

So I left the IMC with intermediate gains so that it keeps locking and unlocking.

VCO (overall) iMC gain of -32, FSS common gain 3, and the FAST gain 20. I believe MC2tickle is ON too.

SURFs had trouble connecting paola to martian via wireless.
Of course, it requires a fixed IP but it had not it yet. So I went to chiara and gave 192.168.113.110 as "paolawl". Note that the wired connection has .111 and it is "paola".

Followed the instruction on http://nodus.ligo.caltech.edu:8080/40m/14121

As directed by Gautam, I have set up one script- interact.py (at /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/interact.py) to perform the following two tasks:

1. View the GigE feed for a fixed period of time.
2. Record the GigE feed for a fixed amount of time.

Steps to view GigE feed for a fixed amount of time:

1. Run the following commands in the terminal to navigate to the concerned directory and then view the feed
   1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
   2. python interact.py --path_to_config <path_config> --mode 0 --view_time <viewing_time>, where
      1. path_config: full path to configuration file, defaults to /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini if --path_to_config is not used
      2. viewing_time: time in seconds for which the feed is to be displayed. The server is closed  after this time and the window freezes and can be manually closed.
   3. Exiting the feed in between: The script terminates automatically after the specified time. To terminate the feed in between, close the window manually using the x icon the top right. This makes sure that the server is correctly closed. If closed using the Ctrl-C command in the terminal, the server is left running and any attempt to unwittingly set up another results in an error (see Attachment #1). In this case, the server and client processes needs to be identified manually and killed. I have used the following steps
      1. ps -eaf | grep server, then identify the PID for the python camera_server.py process
      2. kill PID
      3. similarly for the client file

Steps to record the GigE feed for a fixed amount of time:

I tried to look for elegant solutions that wouldn't require editing the code that Jon wrote and stumbled upon this useful bit of information but ended up deciding that it was just easier to change the camera_client_movie.py (/opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/camera_client_movie.py). It can still be run as previously described, where video recording is terminated by using Ctrl-C. Steps to record for a fixed period of time are

1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
2. python interact.py --path_to_config <path_config> --mode 1 --save_time <recording_time> --file_name filename, where\
   1. path_config: full path to configuration file, defaults to /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini if --path_to_config is not used
   2. recording_time: time in seconds for which the feed is to be saved. No video is displayed during this time.
   3. filename: full path to the file where the video is to be saved. Overwrites any existing files.

I'll make aliases for these to make the whole process more user friendly. I'm halting this for now and will discuss what else needs to be done once Gautam gets back.

Regarding the autolocker: I spoke to Aaron today and as he is in tomorrow, I'll ask him about the burt files and the ideal configuration.

I'm also starting with GANs now.

Continuing this investigation of the IMC, today I am getting familiar with the PMC and FSS. I'd like to measure the frequency noise of the PSL referenced to the PMC.

I checked that the PSL shutter is off, so no light reaches the IMC.

I'm not really sure what I'm looking for on the FSS boxes. I found a few documents to guide:

• On the tabletop testing of FSS for LIGO
• 40m wiki

I ran the FSS autolock script from C1PSL_FSS, nothing obvious changes when I do so. The FSS error signal (which I think is PSL-FSS_MIXERM) is flatlined, and the RC-RF_PD has no LO (PSL-FSS_LOCALC is nan).

Rich dropped by at around 3:00 PM today and picked up the VCO in Attachment #1 and left the note in Attachment #2 on Gautam's desk with the promise of bringing it back soon.

don't need to lock - make sure the 4 OSEMs are centered on the camera field just as we have for the arm cavity mirrors

Today, Rana asked me to work on improving simulations based on the ideas we discussed last week. As of the previous elog the simulation accomodated only

1. Simulation of Gaussian beam spot.
2. Arbitrary motion.

Today, I added the simulation of point scatterers.

What?

The image on the sensor (camera) is produced in roughly the following steps.

1. Motion of the Gaussian beam on the optic (X,Y coordinates) which is what has been simulated so far.
2. Reflection from the surface of the optic which can be modeled using knowledge of the BRDF has not been included as of this elog as I wish to do a little more reading before doing so.
3. Reflection from point scatterers (dust particles burnt into the optic surface by the laser and so forth) which are characterised as peaks (impulses) in the TIS vs position plot. The laser beam is incident nearly normally on the optic and this behaviour is independent of the angle of observation. This is what has been added to the simulation.

How?

1. Increased the frame resolution to 720 x 480.
2. Defined an array of the same size and set values of at most "num_scatter" number of points at random positions to values determined randomly between 1 and "scatter_amp" + 1 where scatter_amp is non-negative.
3. Multiplied the resulting array by the resulting Gaussian beam. The motivation was to imitate the bright specks obtained on various camera feeds in the lab. Physically, this also implies normal incidence and normal observation which is not the real case at all. I shall add these features in a day or two.

Herewith, in attachments #1, #2, #3 I am attaching videos obtained by varying scattering amplitude and number of scattering points in a vain attempt to reproduce this data. I shall work more on this simulation on Friday.

Scripting stuff:

1. Previous elogs detail how to take gige images at various exposure times. I am still waiting on Kruthi to use the script.
2. Tomorrow I shall work on the scripting software to interact with the GigE and take video for a fixed duration etc. I shall also begin working on a script to autolock the PMC based on what Rana showed me on Monday. I will also take a look at the the contents of this elog and try to pick up from there. I hope to make significant progress by the next lab meeting.

Neural network stuff:

GANs for simulation:

1. Other than putting the physics into simulation i.e the first portion of this elog, GANs can be trained to generate images similar to the original data. I am unfamiliar with training GANs and the various tricks that are used specifically for them. I will do a bit of reading and make an update by Friday. As of now, the data I plan to use is this and I will train it using the GTX 1060 on my machine.

Networks for beam tracking:

1. I will use the architectures suggested in this work with a few modifications. I will use MSE loss function, Adam optimizer and my local GPU for training.

I'm attaching a picture of the screen. I just positioned the enclosure by turning it a bit and I suppose we can see the mirror inside the vacuum now (the MC2 is still not locked). 

[Koji, Kruthi]

Yesterday, Koji helped me clean all the optics that are being used for the setup. We tried aligning the cameras with the previous configuration we had, but after connecting the analog camera cables there wasn't much room to align the beam splitter. Today, I tried a different configuration and tested the alignment of analog camera, GigE, beam splitter and the mirror using a laser beam [pictures attached]. But the MC2 isn't locked to test if the whole setup is actually aligned with the mirror inside the vacuum. 

Also, with this setup, just by using posts of different lengths with the middle 90º-post-clamp, we will be able to move all the components. This way, we can easily image the beam spot in all the cavities.

I drew out some idea of how we might use a single OMC to clean both paths of the BHD after mixing, without being susceptible to polarization-dependent effects within the OMC. Basically, can we send the two legs of the BHD into the OMC counterpropagating. I've attached a diagram.

I think one issue would be scattered light, since any backscatter directly couples into the counterpropagating mode, and thus directly to the PD. However, unless the polarization of the scattered light rotates it would not scatter back to the IFO. And, since the LO and signal mix before the OMC, this scattered light would not directly add phase noise.

Maybe more problematic would be that if the rejection at the PBS (or the polarization rotation) isn't perfect, light from the LO directly couples into the dark port. Can we get away with a Faraday isolator before the OMC? 

Diagram attached.

[Kruthi, Milind]

Today, with Milind's help, I installed the analog camera into the MC2 enclosure [picture attached]; but it is not yet focused. We replaced the bulky angular bracket with a simple one, this saved a lot of space inside and it's easier to align other components now. I'll finish setting it up tomorrow.

Telescope design for MC2:  Instead of using two 3" long stackable lens tubes (SM2L30), we can use one 3" lens tube with an adjustable lens tube (SM2V10), as shown in the picture. This gives a flexibility to change the focal plane distance by 1" and also reduces the overall length of telescope from 9 inches to 6-7 inches. I decided to use two 150mm biconvex lens instead of a combination of 150mm and 250mm lenses, as the former combination results in lower focal plane distance for a given distance between the lenses.

Specifications of current telescope system (for future reference):

With the above telescope, assuming the MC2 mirror to be at a distance of approx 75cm, the focal plane distance will range from 7.9cm to 8.1cm. Using the adjustable lens tube, we can further make the fine adjustment.

Today, Rana had me key the PSL crate.

1. Locating the rack: the crate is 1X1. This link provides details of the locations and functions of the racks.
2. Keying the crate: the key is located at the bottom of the rack (in this case). Keying it requires one to turn the key through 90 degrees (anti clockwise facing the rack) and back to to the original position.

Locking the PMC:

1. Accessing the medm screen for the PMC: open a new terminal and use the command sitemap. This should open up the sitemap medm screen. Click on the PSL button and then select C1PSL_PMC from the dropdown that is produced. This opens up a medm screen similar to that in Attachment #1.
2. The correct toggling: The keying of the crate sometimes scrambles the settings on the medm screen. Rana and I performed extensive toggling of the buttons and concluded that the combination in Attachment #1 ought to be the correct one.
3. Locking the PMC: The state of the PMC was deduced by observing CH01 on monitor 7. When not locked, there is no observable bright spot. At this point the "Input Offset (V)" slider is set to zero and the "Servo Gain Adjust (dB)" slider is set to minimum. To obtain lock, complete step 2 and then move the "DC Output Adjust (V)"  slider (at the bottom left on the screen) around rapidly while looking for a bright spot. On observing such a spot on the monitor, release the slider and quickly increase the "Servo Gain Adjust (dB)" slider to around 15 dB. Higher gain values produce a bright spot on CH02 as well which vanishes (almost) on decreasing the gain to the aforementioned value.

Steps to take snapshots using GigE at different exposures [Instructions for Kruthi]:

1. Setup C1-CAM-ETMX.ini (/opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini) appropriately. The parameter Number of Snapshots determines how many snapshots will be taken at any given exposure. Set Name Overlay, Time Overlay, Calculation Overlay, Calculations (if using very low values of exposure) and Auto Exposure to False. Ensure that that the IP address of the Camera in use and that in the configuration file match.
2. Launch a server using the following commands (as described in elog 14649)
   1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
   2. python camera_server.py -c C1-CAM-ETMX.ini
3. Open another terminal in the same directory and then run the following command
   1. python exposure_variation.py --minval <minval> --maxval <maxval> --step <step> where
      1. minval: lower bound of range of exposure values, defaults to 150
      2. maxval: upper bound of range of exposure values, defaults to 100000
      3. step: step size of variation in the range [minval, maxval], defaults to 2000

The python script takes in the above parameters and then takes snapshots by setting the exposure to values starting at minval and going upto maxval incrementing by step at each turn. This uses a simple for loop and is nothing elaborate.

A few unrelated updates:

1. On a sidenote, I installed Sublime Text editor on rossa following the instructions at this site (check install using yum section). Further, I have also installed miniconda but did not set it up fully as I was in a rush and did not want to disturb any previously set up environment variables.
2. I have cloned Gabriele's repository and am trying to get it to work on my system. As Gautam has pointed out that the end goal is to get stuff working on the lab machines, I will sharea .yml file with the necessary environment details upon completion.
3. I will upload details of how I am going to construct the two learning tasks that Rana, Gautam and I discussed in a day or two including details of the use of simulation data for training data in the absence of real data (until Kruthi is done setting up the GigE) which Gautam suggested I do to speed things up.

I managed to fit all the parts into the cylindrical enclosure without having to drill a hole in the enclosure to mount the analog camera (pictures attached); thanks to Koji for helping me find some fancy mechanical components (swivel post clamps, right angle post clamps and brackets). On Thursday, with Chub's help, I took a look at all the current analog camera positions with respect to the cylindrical enclosures. I think this setup gives me enough flexibility to align the components, as necessary, to be able to image the test masses/mirrors in all the cavities. I'll set it up for MC2 tomorrow.

As per Gautam's request, I looked at the IMC situation.

Locking path

• Acquisition: IMC IN1 Gain +4 (nominal), Boost 0, VCO Gain (-32), FSS Common +6 (nominal), FSS FAST +20
  This is too low gain. So oscillate VCO Gain between -32 and ~0 until TEM00 lock is acquired
• Once lock is acquired, bring the VCO gain to +11 (new nominal), and increase the FSS FAST to +23 (new nominal). Change the IMC BOOST to 3 (nominal)

Diagnosis

• The PMC servo gain was checked. The control signal monitor for the PMC actuation was hooked up to SR785. The nominal gain was +18dB. Increasing the gain to 20dB made the servo oscillating. So the nominal gain of +18dB seems still reasonable.
• The status of NOISE EATER was checked. Both the PMC REFL and TRANS were looked at by AG4395A. The power spectrum of them did not change much around the kHz~MHz region. It made the PSD slightly (x2~3) improved below 1kHz. I also did not recognize the relaxation oscillation peak. So I could not figure out where to see. NOISE EATSER was on and is still on.
• IFO Modulation Freq: I took this chance to look at the IMC absolute length using the peak at 3.6MHz. The TP1A output of the IMC servo board was hooked up to AG4395A.
  The new FSR of the IMC (and thus the modulation frequency for the IFO) is 11.066275MHz (instead of the previous 11.066209MHz).
  This corresponds to 0.16mm difference in the roundtrip length.
• (*Still working) IMC SERVO configuration:
  • FAST 25 (nominal) sometimes invoke the oscilattion. 24 has gain peaking ~30kHz. There is a big line peak at 35kHz so wanted to avoid the servo bump (PZT-EOM cross over). So decided to use 23dB. (This is not optimal for the CM servo as we need as much as bandwidth for CM servo.)
  • IMC VCO GAIN (bad name. this is actually overall output gain for IMC) was increased from the nominal 7 to 11. Increasing this above 11 makes the servo oscillating at ~200kHz.
• (*Still working) Measured power spectrum of the error signal. Too many line peaks.
• (*Still working) Single trigger observation: Oscilloscope monitoring started from 35kHz going up and ~20kHz oscillation +/-6V of the IMC servo output was observed. Could not capture good data for this. Try the other day.

I'll complete the entry later.

Summary:

I did some more calculations based on our discussions at the meeting yesterday. Posting preliminary results here for comments.

Details:

Attachment #1 - Schematic illustration for the scattering scenarios. For all three scenarios, we would like for the scattered field to be lower than unsqueezed vacuum (safety factor to be debated).

Attachment #2 - Requirements on a fraction  of the counter-propagating resonant mode of the OMC scattering back into the antisymmetric port, as a function of RIN and phase noise on this field (y-axis) and amount of field (depends on the amount of contrast defect light which can become resonant in the counter propagating mode). I don't encode any frequency dependence here.

Attachment #3 - Requirements on the direct scatter from the arm cavity resonant field (assumed to dominate any contribution from the PRC) onto the OMC DCPDs, for some assumed phase noise (y-axis) and fraction of the field that makes it onto the OMC DCPDs. This is a pretty stringent requirement. But the probability is low (it is the product of three presumably small numbers, (i) probablity of the beam scattering out of the TEM00 mode, (ii) BRDF of the scattering surface, (iii) probability of scattering back towards the DCPDs), so maybe feasible? I didn't model any RIN on this field, which would be an additional noise term to contend with. The range of the y-axis was chosen because I think these are reasonable amplitudes for chamber wall  / other scattering surface motion at acoustic frequencies.

[Koji, Milind]

Today I ran into the following errors:

1. Inability to access the EPICS channels using the commands caget and caput and thus the generation of a blank medm screen (error in Attachment #1) when simultaneously running the code in camera_server.py (/opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/camera_server.py).
2. Inability to run camera_server.py code with an active medm screen with a "... failed to read <EPICS channel>" error.

Therefore, Koji and I took a look at it and putting our faith in Gautam's hunch from elog 13023, we walked down to rack 1Y1 and keyed it. Following this, all the functionality previously described was restored! Koji then took a look at all the channels handled by this machine and bestowed upon me the permission to key the crate should I lose control of the GigE again.

Thanks! It does indeed do the trick! With that I was able to

1. Obtain current exposure value using the terminal command caget C1:CAM-ETMX_EXP
2. Set exposure value using the terminal command caput C1:CAM-ETMX_EXP <desired_exposure_value>

Further, a quick look at the camera server code in /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/camera_server.py revealed that the script expects the details of "Number of Snapshots" in "Camera Settings" in the configuration file i.e in C1-CAM-ETMX.ini at ( /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon/C1-CAM-ETMX.ini) which wasn't present before. Adding this parameter to the config file allows one to take a snapshot using the medm screen. Infact, unlike as described in this elog, I was able to start the server and client as described in elog 14649, and then obtain snapshots using the terminal command  caput C1:CAM-ETMX_SNAP 1.

caget/caput probably does the job.

Figured out how to get/grab frames by looking at the pypylon documenation as that turned out to be easier than modifying Jon's code. Still not sure about how to modify the exposure time (other than using the pylon app, the only technique I know so far is to adjust the exposure manually on the medm screen and then run the scripts as described in the previous elog). I will figure that out tomorrow and make a script suitable for Kruthi's usage (obtain a bunch of images with different exposure times). I will also try and integrate the video saving and streaming code into this and have a neat little script set up asap.

I briefly managed to lock the IMC today - it stayed locked for ~10 minutes. Attachment #1 shows spectra of a few error and control signals for today's lock, and from a stretch yesterday before the problems surfaced*. The 60 Hz lines are much bigger, and MC_F signals broadband excess noise above a few Hz. I suspect a problem somewhere in the electronics.

*I confess the comparison isn't entirely valid because I had to tweak the FSS FAST gain from its nominal value of 22 to 25 in order to get the PC drive RMS down to the ~1.5V level. At the nominal gain setting, with the laser frequency locked to the cavity length, the PC Drive RMS was ~4 V. Still, indicative of something being off in the electronics.

​Summary:

Attachment #1 shows the RIN and phase noise requirements for the 40m BHD for measuring Ponderomotive squeezing.

Some details:

1. The interferometer topology is not systematically optimized - I just picked values which are likely close to what we will eventually choose. Namely, , , , , . Nevertheless, I think these requirements will not change by more than 30% for changes to the interferometer config.
2. The requirements are evaluated using the following criterion: assuming that the dominant noises are (i) coil driver at mid-frequencies and (ii) quantum noise at high frequencies, what do the RIN and phase noise on the LO have to be such that the equivalent displacement noise is a factor of 10 below? I opted for a safety factor of 10, this can be relaxed. 
3. An unknown is how much contrast defect light we will end up having due to the mismatch between arms. I assumed a few representative values.
4. The calculations were done analytically. This paper provides a good summary of the relations - although my RIN requirement is more stringent because of the safety factor of 10, and phase noise requirement is less stringent (despite the same safety factor) because we plan to read out at nearly the amplitude quadrature.
5. Since we are discussing the possibility of delivering the LO field using a fiber-coupled pickoff of the laser prior to RF sidebands being added, these requirements do not benefit from passive filtering from the cavity transfer functions. Consequently, the requirements are pretty challenging I think.

Conclusions:

1. The RIN requirement looks very challenging - we will need a shot noise limited ISS with 100 mW DC sensing light, and will likely have to relax the safety factor depending on how much contrast defect light we end up having. This actually sets some requirement on the amount of filtering we need from the OMC (next step).
2. The phase noise requirement also looks very challenging - I need to look up what is possible with the double-pass through fiber technique.

​Next steps:

1. Evaluate the pointing stability requirement on the LO field (IFO output is filtered by the OMC).
2. We still need to think of a control scheme for the LO phase - likely, I think we will need a suspended optic between the fiber collimator delivering the light to the BHD setup with some kind of length actuation capability. 
3. Numerical validation of this analytic study. I believe Finesse is still missing some capabilities that allow us to calculate these couplings, but I'll ask the experts to be sure.
4. Build up the requirements on the OMC cavity:
   • Backscatter requirement (related = OFI isolation requirement, relative length noise between SRM and OMC, OFI and SRM). Does the OFI also have to be suspended?
   • Filtering requirement
   • Pointing stability requirement
   • Length noise requirement 

Chub and I are trying to figure out a way to co-mount GigE into the existing cylindrical enclosure. I'm attaching a picture of the current setup that is being used for imaging MC2. As of now, I have thought of 3 possible setups (schematics attached); but I don't know how feasible they are. Let us know if you have any other ideas.

Update: The setup 3 would require us to use the 52cm long enclosure. It has a long breadboard welded to it, which makes it very convienient, but the whole setup becomes quite heavy and it's not that safe to install such heavy enclosure on top of the vaccuum system. Also, aligning its components would be more complicated than other setups.

I decided to start with the simple one, therefore, I tried implementing setup 1. Fitting in the analog camera horizontally alongside the telescope turned out to be tricky. Though I did manage to fit it in, it didn't leave any room to change the orientation of the beamsplitter. Like Koji suggested, I'll be trying the setup 2.

I was working with the git repo in the SnapPy_pypylon folder (/cvs/cds/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon) and needed to create a branch. To avoid any confusion, I modified the PS1 variable and that alone in the bashrc file to reflect the git branch so that the prompt now displays the git branch if you enter a repository. This is just an update.

The following steps summarize the steps to setting up and interacting with a GigE camera.

Launching the PylonViewerApp:

1. Open a new terminal using Ctrl + Alt + T on the keyboard.
2. Launch the app using the command pylon.

Using setup python scripts to interact with the GigE (a summary of the steps listed here and here)

1. Connect the GigE camera to the ethernet cable and record its IP address. If the IP address is not printed on the GigE, launch the PylonViewerApp and navigate to the "Tools" dropdown menu and select "pylon IP configurator" to be presented with a list of all connected cameras and their IP addresses.
2. To simply observe the camera feed, open a new terminal and run the following commands:
   1. cd /opt/rtcds/caltech/c1/scripts/GigE/SnapPy_pypylon
   2. python camera_server.py -c C1-CAM-ETMX.ini  (only one config file is present currently and more will be added as more cameras are set up. The "Camera IP" in the  .ini file must match that determined in step 1). This starts the camera server.
3. Open a new tab (Ctrl + Shift + T on the keyboard) in the terminal. You should still be in the same directory as navigated to in step 2.1. Run the following command.
   1. python camera_client.py -c C1-CAM-ETMX.ini
4. This should bring up a feed from the camera. Close at will.
5. To record a video file, repeat steps 1 and 2. Open a new tab as described in step 3. Then run the following command:
   1. python camera_client_movie.py -c C1-CAM-ETMX.ini
6. Enter the full path to the file where you wish to save the movie in the prompt that appears. Use ./your_file_name_here.avi to save the the video in the working directory. Press Ctrl + C to stop recording. The recording can be played by navigating to the location where the recording is stored and running vlc your_file_name_here.avi.
7. To adjust the exposure setting of the camera, open a new terminal and run the command sitemap . This should bring up the medm display in Attachment #1. Click on the Video/Lights button highlighted in red and select GigE. Adjust the exposure value in the next window using the slider before starting the server in step 1. Adjusting the slider once the server is started causes the program to freeze. Also set the Snapshot channel C1:CAM-ETMX_SNAP to off as mentioned in elog 14037.

Upcoming updates:

1. Automatic script to run the above steps.
2. Pre-determining the time duration of the recorded video.
3. Obtaining snapshots.

Since ~ 2 hours ago, the IMC autolocker has not been able to keep the IMC locked. I don't see any obvious trends in the wall StripTool that may point to what's going on. For the brief periods in which a TEM00 mode is locked, the PC Drive RMS level is ~5x what the nominal level is, and while the autolocker is trying to lock the IMC, the PC drive RMS level is hovering around 4V DC, which is high. The PMC Error and Control signal spectra show huge 60 Hz (and harmonics) peaks, and indeed this is visible in the time domain signals as well (on ndscope or on the oscilloscope on the PSL table), but this is not a new feature in the last two hours. Usually, this kind of problem signals that either/both the c1psl or c1iool0 slow machines need to be power-cycled, but I confirmed that both machines are online and telnet-able. Possibilities: (i) some card in the c1psl / c1ioo crates have failed or (ii) something in the MC/FSS electronics chain has failed or (iii) there is a huge amount of excess high-frequency noise from the NPRO.

I am leaving the PSL shutter closed.

no BMP files 

Coupling into the fast axis of the fiber:

The PM couplers I bought require that the light is coupled to the fast axis. The Thorlabs part that Andrew ordered, and which Anjali was using for the MZ experiment, was the opposite configuration, and so the input coupler K6XS mount was rotated to accommodate this polarization. The HWP was also rotated to cut the power into the fiber. I undid these changes. Mode-matching is ~65% (2.42mW/3.70mW) which isn't stellar, but good enough. The PER is ~15dB (ratio of power in fast axis to slow axis is ~40), which I verified using another collimator at the output, and a PBS + two photodiodes. Again isn't stellar but good enough.

EX laser temperature adjustment:

Rana adjusted the temperature of the main laser to 30.61 C. According to the calibration, the EX laser temperature needed to be ~32.8 C. It was ~31.2 C. I made the change by rotating the dial on the front panel of the EX laser controller. Fine adjustment was done using the temperature slider on the ALS screen. With an offset of ~+610 counts, I found a beat at ~80 MHz.

First look at PM beamsplitters:

From my initial test, the beat amplitude was stable to my moving of the fibers . The NF1611 DC monitor reports 2.6 V DC with only the EX light, and 3.15 V DC with only the PSL light. So I should probably cut the PSL power a little to improve the contrast. Assuming the 10 kohm DC transimpedance spec can be believed, this means the expected signal level is 4*sqrt(260uA * 315uA)*700V/A ~0.8 Vpp, and I see ~0.9 Vpp, so roughly things add up (this is actually more consistent with an RF transimpedance of 800V/A, which is maybe not unreasonable). The RF amps for routing this signal to the delay line has been borrowed for the 2um frequency noise experiemnt - I will reacquire it today and check the ALS noise performance.

So overall, I am happy with the performance of the current iteration of the BeatMouth.

[Kruthi, Milind]

Yesterday, we were able to capture some images of objects at a distane of approx 60cm (see the attachment), with the GigE mounted onto the telescope. I think, Johannes had used it earlier to image the ETMX (https://nodus.ligo.caltech.edu:8081/40m/13375). His elog entry doesn't say anything about the focal length of the lenses that he had used. The link to the python code he had used to calculate the lens solution wasn't working. After Gautam fixed it, I took a look at it. He has used 150mm (front lens) and 250mm (back lens) as the focal length of lenses for the calculation. Using the lens formula and an image of a nearby light source, with a very rough measurement, I found the focal lengths to be around 14 cm and 23 cm. So, I'm assuming that the lenses in the telescope are of same focal lengths as in his code, i.e 150mm and 250mm.

To maintain PM fibers all the way through to the photodiode, I had ordered some PM versions of the 50/50 fiber beamsplitters from AFW technologies. They arrived some days ago, and today I installed them in the BeatMouth. Before installation, I checked that the ends of the fibers were clean with the fiber microscope. I also did a little cleanup of the NW corner of the PSL table, where the 1um MZ setup was completely disassembled. We now have 4 non-PM fiber beamsplitters which may be useful for non polarizaiton sensitive applications - they are stored in the glass-door cabinet slightly east of the IY chamber along the Y arm, together with all the other fiber-related hardware.

Anjali had changed the coupling of the beam to the slow axis for her experiment but I ordered beamsplitters which have the slow axis blocked (because that was the original config). I need to revert to this config, and then make a measurement of the ALS noise - if things look good, I'll also patch up the Y arm ALS. We made several changes to the proposed timeline for the summer but I'd like to see this ALS thing through to the end while I still have some momentum before embarking on the BHD project. More to follow later in the eve.

[chub, gautam]

Today, we tried to resuscitate the c1iscaux2 channels by swapping the existing, failed VME crate with the newly freed up crate from c1susaux. In summary, the crate gets power, and the EPICS server gets satrted, but I am unable to switch the whitening gain on the whitening boards. I belive that this has to do with the FAIL LEDs that are on for the XVME-220 units. We were careful to preserve the location of the various cards in the VME crates during the swap. Rather than do a detailed debugging with custom RJ45 cables and terminal emulators, I think we should just focus the efforts on getting the Acromag system up and running.

Our work must have bumped a cable to the c1lsc expansion chassis in the same rack - the c1lsc FE had crashed. I rebooted it using the script - everything came back gracefully.

The vacuum itself was fine - CC1 gauge reported a pressure of 1.3e-5 torr. Note to self: the C1:Vac-CC1_HORNET_PRESSURE channel, which is the analog readback of the Hornet gauge and which is hooked up to an Acromag ADC in the c1auxex chassis, is independent of the status of the c1vac machine, and so can serve as a diagnostic.

However, I was unable to interact with c1vac in any way, the monitor hooked up directly to it was showing a frozen display. So I hard-rebooted the system. It took a few minutes to come back online - but even after 10 minutes of waiting, still no display. In the process of the reboot, several valves were closed off - when the EPICS processes restart, there are momentary instances where the readback channels get an "undefined" value, which prompts the main interlock process to transition to a "SAFE" state. 

Running df -h, I saw that the /var partition was completely full. Maybe this was somehow interfering with the machine running smoothly? Two files in particular, daemon.log and daemon.log.1 were ~1GB each. The contents of these files seemed to be just the readbacks for the caget and caput commands. So I cleared both these files, and now the /var partition usage is only 26%. I also got the display back up and running on the physical monitor hooked up to the c1vac machine's VGA port. Let's see if this has improved the stability situation. The CPU load is still high (~6-7), with most of this coming from the modbus process. Why is this so high? c1susaux has more Acromag units but claims a much lower load of 0.71. Is the CPU of the c1vac machine somehow inferior?

In the meantime, I ssh-ed into c1vac and restored the "Vacuum normal" valve config. During this little escapade, the main volume pressure rose to ~6e-5 torr. It's coming back down smoothly.

Unrelated to this work: we had turned the RGA off for the vent, I powered it back on and re-initialized it this morning.