40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 244 of 344  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  13271   Tue Aug 29 21:36:59 2017 johannesUpdatePSLPSL table auxiliary NPRO

 there is a large (~1 cm) aperture Pockels cell that Frank Siefert was using for making pulses to damage photo diodes. There is a DEI Pulser unit near the entrance to the QIL in Bridge which can drive it.

I'll look for it tomorrow, but I haven't given up on the AOMs yet. I swapped in the ISOMET modulator today and saw the same behavior, both in 0th and 1st order. The fall time is pretty much identical. Gautam saw no such thing in the PSL AOM using the same photodetector.

1st order diffracted                                                          0th order


In the meantime I prepared the fiber mode-matching but realized in the process that I had mixed up some lenses. As a result the beam did not have a waist at the AOM location and thus didn't have the intended size, although I doubt that this would cause the slower decay. I'll fix it tomorrow, along with setting up the fiber injection, beat note with the PSL, and routing the fiber if possible.

  13298   Tue Sep 5 23:13:44 2017 johannesUpdatePSLPSL table auxiliary NPRO

I used Gautam's mode measurement of the auxiliary NPRO (w=127.3um, z=82mm) for the spacing of the optics on the PSL table for the fiber injection and light modulation. As mentioned in previous posts, for the time being there is no Faraday isolator and no broadband EOM installed, but they're accounted for in the mode propagation and they have space reserved if desired/required/available.

The coupler used for the injection is a Thorlabs F220APC-1064, which allegedly collimates the beam from the fiber type we use to 2.4mm diameter, which I used as the target for the mode calculations. I coupled the first order diffracted beam to a ~60m fiber, which is a tad long but the only fiber I could locate that was long enough. The coupling efficiency from free-space to fiber is 47.5%, and we can currently get up to 63 mW out of the fiber.

Tomorrow Steve and I are going to pull the fiber through protective tubing and bring it to the AS port. The next step is then characterizing the beam out of the collimator to match it into the interferometer.

As far as the switching itself is concerned: I confirmed that the exponential decay is still present when looking at the fiber output. I located the DEI Pulser unit in the QIL lab, and also found several more AOMs, including a 200MHz Crystal Technologies one, same brand that the PSL has, where the ringdown was not observed. According to past elogs, with good polarizers we can expect an extinction ratio of ~200 from the Pockels cell, which should be fine, but it's going to be tradeoff switching speed <-> extinction (if the alternate AOM doesn't show this ringdown behavior).

Attachment 1: PSL_IR.pdf
Attachment 2: psl_aux_laser.pdf
  13299   Wed Sep 6 01:09:11 2017 johannesUpdateComputer Scripts / ProgramsNew set of loss measurements

I stumbled upon a faster way to stream data from the TDS3014 oscilloscopes to disk, which speeds the loss measurements up by a lot:  ftp://sprite.ssl.berkeley.edu/pub/sharris/MAVEN_LPW_Preamp/109_TDS3014B_control/tds3014b.py
This convenient(!) set of scripts contains a function that parses the scope's native binary format, for which the acquisition of 1 screenful of data takes <1s as opposed to ~20s, into readable data. I tested it for a bit and concluded that it does what it actually claims to do, but there's one weirdness: It get's the channel offset wrong. However this doesn't matter in our measurement because we're subtracting the dark level, which sees the same (wrong) offset. Other than that it seems okay.

So I started a new set of armloss measurements, and since the data acquisition is now much faster, I was able to squeeze a set of 20 individual measurements for each arm into ~30 minutes. This is the procedure I follow when I take these measurements for the XARM (symmetric under XARM <-> YARM):

  1. Dither-align the interferometer with both arms locked. Freeze outputs when done.
  2. Misalign ETMY + ITMY.
  3. ITMY needs to be misaligned further. Moving the slider by at least +0.2 is plentiful to not have the other beam interfere with the measurement.
  4. Start the script, which does the following:
    1. Resume dithering of the XARM
    2. Check XARM dither error signal rms with CDS. If they're calm enough, proceed.
    3. Freeze dithering
    4. Start a new set of averages on the scope, wait T_WAIT (5 seconds)
    5. Read data (= ASDC power and MC2 trans) from scope and save
    6. Misalign ETMX and wait 5s
    7. Read data from scope and save
    8. Repeat desired amount of times
  5. Close the PSL shutter and measure the PD dark levels

I will write a more comprehensive post describing the data acquisition and processing, let's just look at the results for now: The "uncertainties" reported by the individual measurements are on the order of 1-2 ppm (~1.9 for the XARM, ~1.3 for the YARM). This accounts for fluctuations of the data read from the scope and uncertainties in mode-matching and modulation depths in the EOM. I made histograms for the 20 datapoints taken for each arm: the standard deviation of the spread is a little over 2ppm. We end up with something like:

XARM: 49.3 +/- 2.1 ppm
YARM: 20.3 +/- 2.3 ppm



Attachment 1: XARM_20170905.pdf
Attachment 2: YARM_20170905.pdf
  13301   Thu Sep 7 23:09:00 2017 johannesUpdatePSLPSL table auxiliary NPRO

I brought the DEI Pulser unit and a suitable Pockels cell over from Bridge today (I also found an identical Pockels cell already at the 40m on the SP table, now that I knew what to look for).

I also brought the 200MHz AOM (Crystal Technology 3200-1113) along which can achieve rise times of 10 ns(!). Before I start setting up the Pockels cell I wanted to try this different AOM and look at its switching behavior. It asks for a much smaller beam (<65 um diam.) than what's currently in the path to the fiber (500 um diam.), although it's clear aperture is technically big enough (~1mm diam.). So I still tried, and the result was a somewhat elliptical deflected beam, and the slower decay was again visible after switching the RF input.

I was using the big Fluke function generator for the 200MHz seed signal, a Mini Circuits ZASWA-2-50 switch and a Mini Circuits ZHL-5W-1 amplifier. For the last two I moved two power supplies (+/-5V for the switch and +24V for the amplifier) into the PSL enclosure. I started at low seed power on the Fluke, routing the amplified signal into a 20dB attenuator before measuring it with an RF power meter. The AOM saturates at 2.5W (34 dBm), which I determined is achieved with a power setting on the Fluke of -4 dBm. As expected, this AOM performed faster (~80ns fall time) but I again observed the slower decay.

This struck me as weird and I started swapping components other than the AOM, which I probably should have done before. It turned out that it was the PD I was using (the same PDA10CF Gautam had used for his MC ringdown investigations). When I changed it to a PDA10A (Si diode, 150MHz bandwidth) the slow decay vanished! One last round of crappy screenshots:


Rather than proceeding with the Pockels cell, tomorrow I will make the beam in the AOM smaller and hope that that takes care of the ellipticity. If it does: the AOM can theoretically switch on ~10ns timescale, same for the switch (5-15ns typical), and the amplifier is non-resonant and works up to 500MHz, so it shouldn't be a limiting factor either. If this doesn't work out, we can still have ~100ns switching times with the other AOMs.

  13306   Mon Sep 11 12:40:32 2017 johannesUpdatePSLPSL table auxiliary NPRO

I changed the PSL table auxiliary laser setup to the 200 MHz AOM and put the light back in the fiber. Coupling efficiency is again ~50%, giving us up to about 75 mW of auxiliary laser light on the AS table. The 90% to 10% fall time of the light power out of the fiber when switched off is 16.5 ns with this AOM on the PDA10A, which will be sufficient for the ringdown measurements.

  13307   Mon Sep 11 12:56:40 2017 johannesUpdateComputer Scripts / Programslossmap attempts

I was trying to get a lossmap measurement over the weekend but had some trouble first with the IMC and then with the PMC.

For the IMC: It was a bit too misaligned to catch and maintain lock, but I had a hard time improving the alignment by hand. Fortunately, turning on the WFS quickly once it was locked restored the transmission to nominal levels and made it maintain the lock for longer, but only for several minutes, not enough for a lossmap scan (can take up to an hour). Using the WFS information I manually realigned the IMC, which made locking easier but wouldn't help with staying locked.

For the PMC: The PZT feedback signal had railed and the PMC had been unlocked for 8+ hours. The PMC medm screen controls were generally responsive (I could see the modes on the CCDs changing) but I just couldn't get it locked. c1psl was responding to ping but refusing telnet so I keyed the crate, followed by a burt restore and finally it worked.

After the PMC came back the IMC has already maintained lock for more than an hour, so I'm now running the first lossmap measurements.

  13310   Mon Sep 11 23:31:50 2017 johannesUpdateCameraspost-vent camera capture comparison

The latest pre-unintended vent captures of the test mass face cameras were taken on June 2nd, 2017. Only exposures for ITMYF, ETMYF, and ETMXF exist in /users/sensoray/SensorayCaptures/. I took new captures for those three after locking the arms and having the dither-alignment on for 5+ minutes (exposures were taken after turning the dithering off). The capture script is choking on ITMXF, saying the channel can't lock on. Maybe that's why there's also no reference image for it. Capturing QUAD3, which shows ITMXF in the lower right corner, works, but we don't have a capture for reference. I also recorded dark fields after closing the PSL shutter. Naturally, these don't subtract out as well for the three-month old pictures, but it's actually not terrible and qualitatively one can still compare the subtracted images

Visually, ITMYF and ETMYF do not show a dramatic difference between then and now. ETMXF however, does. To get a numerical estimate for the difference in counts, I worked with the subtracted images and placed an aperture about 1.5x the size of the visible beam blob. I summed up the pixel values inside and subtracted the sum of the pixel values of an equally sized area from the upper left corner of the respective image, which looks free of subtraction artifacts and looks qualitatively similar to the background in the central region.

The pixel sum has gone up by about 50% between the exposures. I still have to do the same for the YARM optics but don't expect such a large discrepancy. Unfortunately we're missing those ITMYF expsures...

All pictures are organized in this format:

Pre-vent exposure Post-vent exposure
Pre-vent subtracted Post-vent subtracted











Attachment 11: ETMXF_pre_sub.bmp
  13330   Mon Sep 25 17:56:33 2017 johannesUpdateComputer Scripts / Programstransmitted power during lossmap

I had to do a reboot + burt restore of c1psl today. It was unresponsive and I couldn't get the PMC to lock. I also had to slightly realign the PMC, and the IMC was too misaligned for the autolocker to catch lock. Adjusting it manually, it was predominantly MC1 PIT that was off. The YARM locked on a 10 mode and had to be aligned manually as well.

I left a script running on Donnatella that tilts ETMX and thus moves the beam on ITMX. I'm monitoring the transmitted power to evaluate sane thresholds for the demodulation offsets in a lossmap measurement. The script will return the IFO to normal after it is done and will take <2 hours to complete (no real clue, but there's no way it takes longer than that for ~50 datapoints).

  13334   Tue Sep 26 22:11:08 2017 johannesUpdateCameraspost-vent camera capture comparison

I configured the remaining GigE-Camera to work on the 40m network. We currently have 3 operational Basler cameras:

The 120gm's have been assigned the IPs  (was already configured) and (freshly configured) and have been labeled accordingly. Note that it was not necessary to connect the out-of-the-box camera directly to a dedicated ethernet adapter whose IP was set manually to 169.254.0.XXX as pointed out in earlier posts - a few seconds after connecting the camera to the control room switch (with PoE adapter to power it) the camera showed up in the configuration software tool which is launched via


and can be assigned a corrected, static IP.

We have a plethora of 2" tubes for the lens assembly, but not a great variety of focal lengths for 2" lenses. Present with the camera gear were two f=250 mm and one f=150 mm 2" lenses with a NIR broadband AR coating

To determine the lens positions relativ to the sensor I assumed that the camera we're setting up looks at its test mass from a distance of 1m. Using the two available focal lengths we can look for solutions which have reasonable lens separations <~10cm and suitable magnification. We primarily want to image the central mirror area onto a 1/4" sized sensor, which can be achieved with a magnification of ~1/8.

I chose a lens separation of 6cm, which gives a theoretical magnification of -.12 and a sensor-lens 2 distance of 7.95 cm. I placed the lenses accordingly in the tubes and checked the focusing with Gautam's help:


It's pretty close to what we would expect. We will do the calibration using the auxiliary laser on the PSL table. For this I temporarily routed a fiber from the PSL enclosure to the SP table. Since the main cable hole is sort of cramped it's going in through a gap near the ceiling instead.  


Attachment 1: lens_distance.pdf
  13348   Mon Oct 2 12:44:45 2017 johannesUpdateCamerasBasler 120gm calibration

Disclaimer: Wrong calibration factors! See https://nodus.ligo.caltech.edu:8081/40m/13391

The two acA640-120gm Basler GigE-Cams have been calibrated. I used the collimated output of a fiber that carried the auxiliary laser light from the PSL table. With a non-polarizing beam splitter some of the light was picked off onto a PD, and I modified the RF amplitude of the AOM drive signal to vary the power coming out of the fiber. The fiber output was directed at a white paper, which was placed 1.06m from the front of the lens tube assembly, which is where the focal plane is. Using the Pylon Viewer App I made sure that the entirety of the beam spot was imaged onto the CCD. Since the camera sensor is 1/4" across, I removed the camera from the lens tube and instead placed the Ophir power meter head at the position of the sensor and measured the power reported versus PD voltage, which turned out to be 1.5 V/uW.

The camera was put back in place and I used the Pypylon package Gautam had stumbled upon to sweep the exposure time from 100us to 10ms at different light power settings including no laser light at all for background subtraction, and rather than keeping the full bitmap data for O(100s) of images I recorded only the quantities

  1. Pixel Max
  2. Pixel Sum
  3. Pixel Mean
  4. Pixel Standard Deviation
  5. Pixel Median

I performed this procedure for both the 152 and 153 cameras and plotted the pixel sum and the pixel max vs the exposure time. All the exposures were taken at a gain setting of 100, which is the smallest possible setting (out of 100-600). To obtain the calibration factor I use the input power Pin=75nW in the 'safe' region 1ms to 10ms where the pixel sum looks smooth and the CCD is reportedly not saturated.

Camera IP Calibration Factor CF 8.58 W*s 7.83 W*s

The incident power can be calculated as Pin =CF*Total(Counts-DarkCounts)/ExposureTime.

Attachment 1: calib_20170930_152.pdf
Attachment 2: calib_20170930_153.pdf
  13354   Tue Oct 3 01:58:32 2017 johannesHowToCamerasCCD calibration

Disclaimer: Wrong calibration factors! See https://nodus.ligo.caltech.edu:8081/40m/13391

The factors were indeed enormously off. The correct table reads:

Camera IP Calibration Factor CF 85.8 pW*s 78.3 pW*s

I did subtract a 'dark' frame from the images, though not in the sense of your point 1, just an exposure of identical duration with the laser turned off. This was mostly to reduce the effect of residual light, but given similar initial conditions would somewhat compensate for the offset that pre-existing charge and electronics noise put on the pixel values. The white field is of course a difference story.

I wonder how close we can get to a white field by putting a thin piece of paper in front of the camera without lenses and illuminate it from the other side. A problem is of course the coherence if we use a laser source... Or we scrap any sort of screen/paper and illuminate directly with a strongly divergent beam? Then there wouldn't be a specular pattern.

I'm not sure I understand your point about the 1.5V/A. Just to make sure we're talking about the same thing I made a crude drawing:

The PD sees plenty of light at all times, and the 1.5V/uW came from a comparative measurement PD<-->Ophir (which took the place of the CCD) while adjusting the power deflected with the AOM, so it doesn't have immediate connection to the conversion gain of silicon in this case. I can't remember the gain setting of the PD, but I believe it was 0dB, 20dB at most.

Attachment 1: gige_calibration.pdf
  13375   Thu Oct 12 01:03:49 2017 johannesHowToCamerasETMX GigE side view

I calculated a better lens solution for the ETMX side view with the simple python script that's attached. The camera is still not as close to the viewport as we would like, and now the front lens is almost all the up to the end of the tube. With a little more playing around there maybe a better way, especially if we expand the repertoire of focal lengths. Using Steve's wonderful camera fixture I put the beam spot in focus. I turned the camera sideways for better use of the field of view, and now the beam spot actually fills the center area of the beam, to the point where we probably don't want more magnification or else we start losing the tails of the Gaussian.

We'll take a serious of images tomorrow, and will have an estimate of the scatter loss by the end of tomorrow.


Attachment 1: IMG_20171011_164549698.jpg
Attachment 2: Image__2017-10-11__16-52-01.png
Attachment 3: GigE_lens_position_helper.py.zip
  13389   Wed Oct 18 11:37:58 2017 johannesHowToCamerasETMX GigE side view at 50 deg

 Telescope front lens to wall distance 25 cm,  GigE camera lenght 6 cm and cat6 cable 2cm

 Atm3,   Existing short camera  can has 16cm  lenght to lexan guard on viewport. Available 2" od periscope tube lenght is 8cm. The one in use 16 cm long.

             Note: we can fabricate a lite cover with tube that would accomodate longer telescope.

             Can we calibrate the AR coated M5018-SW and compare it's performance agains the 2" periscope

             Look at the Edmond Optics 3" od camera lens with AR

Atm1,   Now I can see dust. This is much better. The focus is not right yet.

Atm2,   Chamber viewport wiped and image refocused. Actually I was focusing on the dust.

We don't really have to calibrate the lens, just the CCD, which we've done. It's more about knowing the true aperture size to know how much solid angle you're capturing to infer the total amount of scatter. For our custom lens tubes this is the ID of the retaining ring.

The Edmund Optics lens tube looks tempting, but itcomes at a price. Thorlabs sells lens tubes that offer a more flexibility than what we have right now, so I bought a few different ones, and also more 150mm 2" lenses. This will allow for more compact solutions and offer some in-situ focusing ability that doesn't require detaching the lens tube like now. Should be here in a couple of days, then we'll be able to enclose the GigE camera in the viewport can with a similar field of view we have now.

I also bought a collimation package for the AS port fiber stuff so we can move ahead with the ringdown measurements and also mode spectroscopy.

  13391   Wed Oct 18 15:26:58 2017 johannesHowToCamerasRevision: CCD calibration

The units were still off in my previous post. Here's the corrected, sanity-checked version:

Camera IP Calibration Factor 85.8 +/- 4.3 pW*μs 78.3 +/- 3.9 pW*μs

I estimated the uncertainties based on a linear fit to the data I recorded with 75nW incident on the CCD and assumed a 5% uncertainty in that number. This is just an upper limit, to be safe. I had calibrated the power reading placing the Ophir power meter where the CCD would otherwise be and comparing it to the PD voltage of a picked off beam. In my previous figures the axes were mislabeled, so I reproduce them here:

Using the current camera position I recorded 50 exposures both with and without beam (XARM locked vs PSL shutter closed) and averaged the images to see how much the reading fluctuates. The exposure time was 10 ms, which left the maximum reported pixel value in all exposures below 3800 out of 4096. The gain setting was 100, which is what I used to calibrate the CCDs.

Counts with XARM locked 2.799 +/- 0.027 x107
Counts with shutter closed 3.220 +/- 0.047 x106
Power on CCD 193.9 +/- 2.2 nW
Power scattered into 2π (*) 254 +/- 39 μW
ETMX scatter loss (**) 25.4 +/- 3.9 ppm

(*) I calculated the lens positions to focus at a plane 65cm from the front lens. We're pretty close to that, but I can't confirm the actual distance easily, so I assumed a 5cm error on the distance, which is where most of the error is coming from. This is also assuming uniform scatter.

(**) This is assuming 10W of circulating power

Attachment 1: calib_20170930_152.pdf
Attachment 2: calib_20170930_153.pdf
  13422   Thu Nov 9 15:33:08 2017 johannesUpdateCDSrevisiting Acromag

We probably want to get a dedicated machine that will handle the EPICS channel serving for the Acromag system


This is the machine that Larry suggested when I asked him for his opinion on a low workload rack-mount unit. It only has an atom processor, but I don't think it needs anything particularly powerful under the hood. He said that we will likely be able to let us borrow one of his for a couple days to see if it's up to the task. The dual ethernet is a nice touch, maybe we can keep the communication between the server and the DAQ units on their separate local network.

  13443   Wed Nov 22 00:54:18 2017 johannesOmnistructureComputersSlow DAQ replacement computer progress

I got the the SuperMicro 1U server box from Larry W on Monday and set it up in the CryoLab for initial testing.

The specs: https://www.supermicro.com/products/system/1U/5015/SYS-5015A-EHF-D525.cfm

The processor is an Intel D525 dual core atom processor with 1.8 GHz (i386 architecture, no 64-bit support). The unit has a 250GB SSD and 4GB RAM.

I installed Debian Jessie on it without any problems and compiled the most recent stable versions of EPICS base (3.15.5), asyn drivers (4-32), and modbus module (2-10-1). EPICS and asyn each took about 10 minutes, and modbus about 1 minute.

I copied the database files and port driver definitions for the cryolab from cryoaux, whose modbus services I suspended, and initialized the EPICS modbus IOC on the SuperMicro machine instead. It's working flawlessly so far, but admittedly the box is not under heavy load in the cryolab, as the framebuilder there is logging only the 16 analog channels.

I have recently worked out some kinks in the port driver and channel definitions, most importantly:

  • mosbus IOC initialization is performed automatically by systemd on reboot
  • If the IOC crashes or a system reboot is required the Acromag units freeze in their last current state. When the IOC is started a single read operation of all A/D registers is performed and the result taken as the initial value of the corresponding channel, causing no discontinuity in generated voltage EVER (except of course for the rare case when the Acromags themselves have to be restarted)

Aaron and I set 12/4 as a tentative date when we will be ready to attempt a swap. Until then the cabling needs to be finished and a channel database file needs to be prepared.

  13458   Wed Nov 29 21:40:30 2017 johannesOmnistructureComputersSlow DAQ replacement computer progress

[Aaron, Johannes]

We configured the AtomServer for the Martian network today. Hostname is c1auxex2, IP is Remote access over SSH is enabled.

There will be 6 acromag units served by c1auxex2.

Hostname Type IP Address
c1auxex-xt1221a 1221
c1auxex-xt1221b 1221
c1auxex-xt1221c 1221
c1auxex-xt1541a 1541
c1auxex-xt1541b 1541
c1auxex-xt1111a 1111

Some hardware to assemble the Acromag box and adapter PCBs are still missing, and the wiring and channel definitions have to be finalized. The port driver initialization instructions and channel definitions are currently locally stored in /home/controls/modbusIOC/ but will eventually be migrated to a shared location, but we need to decide how exactly we want to set up this infrastructure.

  • Should the new machines have the same hostnames as the ones they're replacing? For the transition we simply named it c1auxex2.
  • Because the communication of the server machine with the DAQ modules is happening over TCP/IP and not some VME backplane bus we could consolidate machines, particularly in the vertex area.
  • It would be good to use the fact that these SuperMicro servers have 2+ ethernet ports to separate CDS EPICS traffic from the modbus traffic. That would also keep the 30+ IPs for the Acromag thingies off the Martian host tables.
  13463   Mon Dec 4 22:06:07 2017 johannesOmnistructureComputersAcromag XEND progress

I wired up the power distribution, and ethernet cables in the Acromag chassis today. For the time being it's all kind of loose in there but tomorrow the last parts should arrive from McMaster to put everything in its place. I had to unplug some of the wiring that Aaron had already done but labeled everything before I did so. I finalized the IP configuration via USB for all the units, which are now powered through the chassis and active on the network.

I started transcribing the database file ETMXaux.db that is loaded by c1auxex in the format required by the Acromags and made sure that the new c1auxex2 properly functions as a server, which it does.


  • Need to calibrate the +/- 10V swing of the analog channels via the USB utility, but that requires wiring the channels to the connectors and should probably be done once the unit sits in the rack
  • Need to wire power from the Sorensens into the chassis. There are +/- 5V, +/- 15V and +/- 20V present. The Acromags need only +12V-32V, for which I plan to use the +20V, and an excitation voltage for the binary channels, for which I'm going to wire the +5V. Should do this through the fuse rails on the side.
  • The current slow binary channels are sinking outputs, same as the XT1111 16-channel module we have. The additional 4 binary outputs of the XT1541 are sourcing, and I'm currently not sure if we can use them with the sos driver and whitening vme boards that get their binary control signals from the slow system.
  • Confirm switching of binary channels (haven't used model XT1111 before, but I assume the definitions are identical to XT1121)
  • Setup remaining essential EPICS channels and confirm that dimensions are the same (as in both give the same voltage for the same requested value)
  • Disconnect DIN cables, attach adapter boards + DSUB cables
  • Testing



[Aaron, Johannes]

We configured the AtomServer for the Martian network today. Hostname is c1auxex2, IP is Remote access over SSH is enabled.

There will be 6 acromag units served by c1auxex2.

Hostname Type IP Address
c1auxex-xt1221a 1221
c1auxex-xt1221b 1221
c1auxex-xt1221c 1221
c1auxex-xt1541a 1541
c1auxex-xt1541b 1541
c1auxex-xt1111a 1111

Some hardware to assemble the Acromag box and adapter PCBs are still missing, and the wiring and channel definitions have to be finalized. The port driver initialization instructions and channel definitions are currently locally stored in /home/controls/modbusIOC/ but will eventually be migrated to a shared location, but we need to decide how exactly we want to set up this infrastructure.

  • Should the new machines have the same hostnames as the ones they're replacing? For the transition we simply named it c1auxex2.
  • Because the communication of the server machine with the DAQ modules is happening over TCP/IP and not some VME backplane bus we could consolidate machines, particularly in the vertex area.
  • It would be good to use the fact that these SuperMicro servers have 2+ ethernet ports to separate CDS EPICS traffic from the modbus traffic. That would also keep the 30+ IPs for the Acromag thingies off the Martian host tables.
  13464   Thu Dec 7 11:14:37 2017 johannesHowToComputer Scripts / ProgramsLots of red on the FE status screen

Since we're getting ready to put the replacement slow DAQ for c1auxex in I wanted to bring the IFO back to operating condition after the PMC hasn't been locked for days. Something seems wrong with the CDS system though, many of the frontent models have red background and don't seem to be responsive. I followed the instructions laid out in https://wiki-40m.ligo.caltech.edu/Computer_Restart_Procedures.

In the attached screenshot, initially all c1ioo models were red, and on c1iscex only c1x01 was blue, the other ones red. I was able to ssh into both machines and tried to restart indivitual models, which didn't work and instead turned their background white. Still following the wiki page, I restarted both machines but they don't respond to pinging anymore and thus I cannot use ssh to reach them. Not sure what to do, I also rebooted fb over telnet.

So far I couldn't find any records of how to fix this situation.

Attachment 1: 22.png
  13466   Thu Dec 7 15:46:31 2017 johannesHowToComputer Scripts / ProgramsLots of red on the FE status screen

[Koji, Johannes]

The issue was partially fixed and the interferometer is in workable condition now.

What -probably- fixed it was restarting the dhcp server on chiara

sudo service isc-dhcp-server restart

Afterwards the frontends were restarted one by one. SSH access was possible and the essential models for IFO operation were started.

c1iscex reported initially that no DAQ card was found, and inside the IO chassis the LED indicator strip was red. Turning off the machine, checking the cables and rebooting fixed this.

Attachment 1: 04.png
  13468   Thu Dec 7 22:24:04 2017 johannesOmnistructureComputersAcromag XEND progress


  • Need to calibrate the +/- 10V swing of the analog channels via the USB utility, but that requires wiring the channels to the connectors and should probably be done once the unit sits in the rack
  • Need to wire power from the Sorensens into the chassis. There are +/- 5V, +/- 15V and +/- 20V present. The Acromags need only +12V-32V, for which I plan to use the +20V, and an excitation voltage for the binary channels, for which I'm going to wire the +5V. Should do this through the fuse rails on the side.
  • The current slow binary channels are sinking outputs, same as the XT1111 16-channel module we have. The additional 4 binary outputs of the XT1541 are sourcing, and I'm currently not sure if we can use them with the sos driver and whitening vme boards that get their binary control signals from the slow system.
  • Confirm switching of binary channels (haven't used model XT1111 before, but I assume the definitions are identical to XT1121)
  • Setup remaining essential EPICS channels and confirm that dimensions are the same (as in both give the same voltage for the same requested value)
  • Disconnect DIN cables, attach adapter boards + DSUB cables
  • Testing

Getting the chassis ready took a little longer than anticipated, mostly because I had not looked into the channel list myself before and forgot about Lydia's post which mentions that some of the switching controls have to be moved from the fast to the slow DAQ. We would need a total of 5+5+4+8=22 binary outputs. With the existing Acromag units we have 16 sinking outputs and 8 sourcing outputs. I looked through all the Eurocrate modules and confirmed that they all use the same switch topology which has sourcing inputs.

While one can use a pull-down resistor to control a sourcing input with a sourcing output,

pulling down the MAX333A input (datasheet says logic low is <0.8V) requires something like 100 Ohms for the pull down resistor, which would require ~150mA of current PER CHANNEL, which is unreasonable. Instead, I asked Steve to buy a second XT1111 and modified the chassis to accomodate more Acromag units.

I have now finished wiring the chassis (except for 8 remaining bypass controls to the whitening board which need the second XT1111), calibrated all channels in use, confirmed all pin locations via the existing breakout boards and DCC drawings for the eurocrate modules, and today Steve and I added more fuses to the DIN rail power distribution for +20V and +15V.

There was not enough contingent free space in the XEND rack to mount the chassis, so for now I placed it next to it.

c1auxex2 is currently hosting all original physical c1auxex channels (not yet calc records) under their original name with an _XT added at the end to avoid duplicate channel names. c1auxex is still in control of ETMX. All EPICS channels hosted by c1auxex2 are in dimensions of Volts. The plan for tomorrow is to take c1auxex off the grid, rename the c1auxex2 hosted channels and transfer ETMX controls to it, provided we can find enough 37pin DSub cables (8). I made 5 adapter boards for the 5 Eurocrate modules that need to talk to the slow DAQ through their backplane connector.

  13469   Fri Dec 8 12:06:59 2017 johannesOmnistructureComputersc1auxex2 ready - but need more cables

The new slow machine c1auxex2 is ready to deploy. Unfortunately we don't have enough 37pin DSub cables to connect all channels. In fact, we need a total of 8, and I found only three male-male cables and one gender changer. I asked Steve to buy more.

Over the past week I have transferred all EPICS records - soft channels and physical ones - from c1auxex to c1auxex2, making changes where needed. Today I started the in-situ testing

  1. Unplugged ETMX's satellite box
  2. Unplugged the eurocrate backplane DIN cables from the SOS Driver and QPD Whitening filter modules (the ones that receive ao channels)
  3. Measured output voltages on the relevant pins for comparison after the swap
  4. Turned off c1auxex by key, removed ethernet cable
  5. Started the modbus ioc on c1auxex2
  6. Slow machine indicator channels came online, ETMX Watchdog was responsive (but didn't have anything to do due to missing inputs) and reporting. PIT/YAW sliders function as expected
  7. Restoring the previous settings gives output voltages close to the previous values, in fact the exact values requested (due to fresh calibration)
  8. Last step is to go live with c1auxex2 and confirm the remaining channels work as expected.

I copied the relevant files to start the modbus server to /cvs/cds/caltech/target/c1auxex2, although kept local copies in /home/controls/modbusIOC/ from which they're still run.

I wonder what's the best practice for this. Probably to store the database files centrally and load them over the network on server start?

  13470   Fri Dec 8 23:31:31 2017 johannesFrogsASSc1ass slow channel offloading scripts with small

While staring at epics records all day I noticed something about the PIT/YAW offset sliders and ASS offset offloading to slow channels scripts that I'm not sure others are aware off, so I'll briefly discuss it in this post.

The PIT and YAW sliders directly control soft channels that are hosted on the slow machine. Secondary epics records disentangle them for the individual coils:

  • UL = PIT+YAW
  • LL = -PIT+YAW
  • UR = PIT-YAW
  • LR = -PIT-YAW

These channels are the direct input for the physical output channels that generate the control voltage.

The fast channels for PIT and YAW have a numerical correction factor built in that accounts for differences between the OSEMs, but the slow channels don't. This means that the slow PIT/YAW controls are not entirely orthogonal but have crosstalk on the order of 10 percent. This in itself is not that dramatic, however the offload offsets scripts for the dither alignment use the fast PIT/YAW values as inputs, which represent the necessary adjustments to the OSEMs only after the individual correction factors have been applied. The offloading to slow knows nothing of this calibration difference between the OSEMs. The result is that there is a ~10 percent of the offset correction error on the mirror alignment AFTER offloading. This will of course converge after a few iterations, but in any case it is recommendable to run the dither alignment again after offloading and not offload the new offsets to the fast channels.

  13471   Wed Dec 13 09:49:23 2017 johannesUpdateASSwiring diagram

I attached a wiring schematic from the slow DAQ to the eurocrate modules. Of these, pins 1-32 (or 1A-16C) and pins 33-64 (17A-32C) are on separate DSub connectors. Therefore the easiest solution is to splice the slow DIO channels into the existing breakouts so we can proceed with the transition. This will still remove a lot of the current cable salad. For the YEND we can start thinking about a more elegant solution (For example a connector on the front panel of the Acromag chassis for the fast DIO) now that the problem is better defined.

Attachment 1: 1Y9.pdf
  13473   Thu Dec 14 00:32:56 2017 johannesUpdateASSAcromag new crate; c1auxex2 configured as gateway server for acromag

This splicing in of fast binary channels we discussed at yesterday's and today's meetings is getting messy with the current chassis. Cleaning up the cable mess was a key point, so I got a 4U height DEEP chassis from Rich and drew up a front panel for a modular approach that we can use at the other 40m locations as well. The front panel will have slots for smaller slot panels to which we can mount the breakout boards as before, so all the wiring that I've done can be transfered to this design. If some new connector standard is required it will be easy to draw a new slot panel from a template, for now I'll make some with two DSub37 and IDC50. Since this chassis is so huge it will have ample space for cross-connects.

I also moved the communication of c1auxex2 with the Acromag units off the martian network, connecting them with a direct cable connection out of the second ethernet port. To test if this works I configured the second ethernet port of c1auxex2 to have the IP address and one of the acromag units to have, and initialized an IOC with some test channels. Much to my surprise this actually worked straight out of the box, and the test channels can be accessed from the control room computers without having a direct ethernet link to the acromag modules. huzzah!

Steve: it would be nice to have all plugs- connectors lockable


Attachment 1: fp_mod_4U.pdf
Attachment 2: IMG_20171213_171541850_HDR.jpg
  13478   Thu Dec 14 23:27:46 2017 johannesUpdateDAQaux chassis design

Made a front and back panel and slot panels for DSub and IDC breakouts. I want to send this out soon, are there any comments? Preferences for color schemes?

Attachment 1: auxdaq_40m_4U_front.pdf
Attachment 2: auxdaq_40m_4U_rear.pdf
Attachment 3: auxdaq_40m_4U_DSub37x2.pdf
Attachment 4: auxdaq_40m_4U_IDC50.pdf
  13479   Fri Dec 15 00:26:40 2017 johannesUpdateCDSRe: CDS recovery, NFS woes

Didn't touch Xarm because we don't know what exactly the status of ETMX is.

The Xarm is currently in its original state, all cables are connected and c1auxex is hosting the slow channels.

  13517   Tue Jan 9 00:07:03 2018 johannesUpdateDAQetmx slow daq chassis

All parts received and assembly near complete, small problem detected because the two DSub connectors are too close together for two cables to fit at the same time. Gautam and I will make some additional slot panels tomorrow using a waterjet cutter, so we can spread the breakout boards out and remedy this.

Fast binary channels need to be spliced into DSub connectors. Aaron is on this. All other, slow connections are already wired from before and have been tested for correct pins on the backplane DIN connectors.


The Acromag modules require only a positive supply voltage between +12 and +30 VDC and draw a maximum of 2.8W at that. This raises the question if we want this supply rail to be regulated or take the raw power from the Sorensens. Consulting with Ben Abbott: The Acromags in LIGO do not operate with regulated power. We could easily accomodate the standard regulator boards D1000217 in the chassis, which is probably a good idea if we want to place any custom electronics inside the chassis in the future, for example for whitening or active lowpass filtering.

  13524   Wed Jan 10 14:17:57 2018 johannesConfigurationComputer Scripts / Programsautoburt no longer making backups

I was looking into setting up autoburt for the new c1auxex2 and found that it stopped making automatic backups for all machines after the beginning of the new year. There is no 2018 folder (it was the only one missing) in /opt/rtcds/caltech/c1/burt/autoburt/snapshots and the /latest/ link in /opt/rtcds/caltech/c1/burt/autoburt/ leads to the last backup of 2017 on 12/31/17 at 23:19.

The autoburt log file shows that the back script last ran today 01/10/18 at 14:19, as it should have, but doesn't show any errors and ends with "You are at the 40m".

I'm not familiar with the autoburt scheduling using cronjobs. If I'm not mistaken the relevant cronjob file is /cvs/cds/rtcds/caltech/c1/scripts/autoburt/autoburt.cron which executes /cvs/cds/rtcds/caltech/c1/scripts/autoburt/autoburt.pl

I've never used perl but there's the following statement when establishing the directory for the new backup:

  $yearpath = $autoburtpath."/snapshots/".$thisyear;
  # print "scanning for path $yearpath\n";
  if (!-e $yearpath) {
    die "ERROR: Year directory $yearpath does not exist\n";

I manually created the /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2018/ directory. Maybe this fixes the hickup? Gotta wait about 30 minutes.

  13525   Wed Jan 10 15:25:43 2018 johannesConfigurationComputer Scripts / Programsautoburt making backups again

I manually created the /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2018/ directory. Maybe this fixes the hickup? Gotta wait about 30 minutes.

It worked. The first backup of the year is now from Wednesday, 01/10/18 at 15:19. Ten days of automatic backups are missing. Up until 2204 the year folders had been pre-emptively created so why was 2018 missing?

gautam: this is a bit suspect still - the snapshot file for c1auxex at least seemed to be too light on channels recorded. this was before any c1auxex switching. to be investigated.

  13529   Wed Jan 10 22:24:28 2018 johannesUpdateDAQetmx slow daq chassis

This evening I transitioned the slow controls to c1auxex2.

  1. Disconnected satellite box
  2. Turned off c1auxex
  3. Disconnected DIN cables from backplace connectors
  4. Attached purple adapter boards
  5. Labeled DSub cables for use
  6. Connected DSub cables to adapter boards and chassis
  7. Initiated modbus IOC on c1auxex2

Gautam and I then proceeded to test basic functionality

  1. Pitch bias sliders move pitch, yaw moves yawyes.
  2. Coil enable and monitoring channels work yes
  3. Watchdog seems to work. yes We set the treshold for tripping low, excited the optic, watchdog didn't disappoint and triggered.
  4. All channels Initialize with "0" upon machine/server script restart. This means the watchdog happens to be OFF, which is good yes. It would be great if we could also initialize PIT and YAW to retain their value from before to avoid kicking the optic. This is not straightforward with EPICS records but there must be a way.
  5. We got the local damping going yes.
  6. There is some problem with the routing of the fast BIO channels through the new chassis - so the ANALOG de-whitening filter seems to be always engaged, despite our toggling the software BIO bits no. Something must be wrongly wired, as we confirmed by returning only the FAST BIO wiring to the pre-acromag state (but everything else is now controlled by acromag) and didn't have the problem anymore. Or some electrical connection is not made (I had to use gender changers on these connectors due to lack of proper cabling)
  7. The switches for the QPD gain stages did not work. no I suspect a wiring problem, since the switching of the coil enables did work.

Arms are locked, have been for ~1hour with no hickups. We will leave it like this overnight to observe, and debug further tomorrow.

  13537   Fri Jan 12 10:02:05 2018 johannesUpdateDAQetmx slow daq chassis

There is some problem with the routing of the fast BIO channels through the new chassis - so the ANALOG de-whitening filter seems to be always engaged, despite our toggling the software BIO bits no. Something must be wrongly wired, as we confirmed by returning only the FAST BIO wiring to the pre-acromag state (but everything else is now controlled by acromag) and didn't have the problem anymore. Or some electrical connection is not made (I had to use gender changers on these connectors due to lack of proper cabling)

The switches for the QPD gain stages did not work. no I suspect a wiring problem, since the switching of the coil enables did work.

Both issues were fixed. In both cases it was two separate causes that prevented them from working.

The QPD gain stage switch software channels were assigned to wrong physical pins of the Acromag, and additionally their DSub cable was swapped with a different one.

The BIO switching had its signal and ground wires swapped on ALL connections, and part of it was also suffering from the cable-mixup.

Both issues were fixed. All backplane signals are now routed through the Acromag chassis.


Gautam and I did notice that occasionally ETMX alignment will start drifting as evident from the OpLev. I want to set up a diagnostic channel to see if the DAC voltages coming from the Acromag are responsible for this.

  13543   Fri Jan 12 19:15:34 2018 johannesUpdateDAQetmx slow daq chassis

Steve and I removed c1auxex from 1X9 today to make space for the DAQ chassis. Steve installed rails for mounting. To install the box I had to remove all cabling, for which I used the usual precautions (disconnect satellite box etc.)

On reconnect c1auxex2 didn't initialize the physical EPICS channels (the 'actual' acromag channels), apparently it had trouble communicating. A reboot fixed this. It's possible that this is because of the direct cable connection without a network switch that exists
between the Acromags and c1auxex. The EPICS server was started automatically on reboot.

Currently the channel defaults need to be loaded manually after every EPICS server script restart with burt. I'm looking for a good way to automate this, but the only compiled burt binaries for x86 (that we can in principle run on c1auxex2 itself) on the martian network are from EPICS version 3.14.10 and throw a missing shared object error. Could be that simply some path variable is missing.

The burt binaries are not distributed by the lscsoft or cdssoft packages, so alternatively we would need to compile it ourselves for x86 or get it working with the older epics version.

  13554   Wed Jan 17 22:44:14 2018 johannesUpdateDAQAcromag checks

This happened because there are multiple ways to scale the raw value of an EPICS channel to the desired output range. In the CryoLab I was using one way, but the EPICS records I copied from c1auxex were doing it differently. Basically this:

DTYP  - Data type -
RVAL Raw value
EGUF Engineering units full scale
EGUL Engineering units low
ASLO Manual scaling factor
AOFF Manual offset
VAL Value

If the "LINR" field is set to "LINEAR", the fields EGUF and EGUL are used to convert the raw value to the channel value VAL. To use them, one needs to enter the voltages that return the maximum and minimum values expected for the given data type. It used to be +10V and -10V, respectively, and was copied that way but that doesn't work with the data type required for the Acromag units. For -some- reason, while the the ADC range is -10V to +10V, this corresponds to values -20000 to +20000, while for the DAC channels it's -30000 to +30000. I had observed this before when setting up the DAQ in the CryoLab, but there we were using "NO CONVERSION", which skips the EGUF and EGUL fields, and used the ASLO and AOFF for manual scaling to get it right. When I mixed the records from there with the old ones from c1auxex this got lost in translation. Gautam and I confirmed by eye that this indeed explains the different levels well. This means that the VMon channelsfor the coils are also showing the wrong voltages, which will be corrected, but the readback still definitely works and confirms that the enable switches do their job.

  1. I take back what I said about the OSEM PD mon at the meeting - there does seem to be to be some overall calibration factor (Attachment #1) that has scaled the OSEM PD readback channels, by a factor of (20000/2^15), which Johannes informs me is some strange feature of the ADC, which he will explain in a subsequent post.


  13555   Wed Jan 17 23:36:12 2018 johannesConfigurationGeneralAS port laser injection

Status of the AS-port auxiliary laser injection

  • Auxiliary laser with AOM setup exists, first order diffracted beam is coupled into fiber that leads to the AS table.
  • There is a post-PMC picked-off beam available that is currently just dumped (see picture). I want to use it for a beat note with the auxiliary laser pre-AOM so we can phaselock the lasers and then fast-switch the phaselocked light on and off.
  • I was going to use the ET3010 PD for the beat note unless someone else has plans for it.
  • I obtained a fixed triple-aspheric-lens collimator which is supposed to have a very small M^2 value for the collimation on the AS table. I still have the PSL-lab beam profiler and will measure its output mode.
  • Second attached picture shows the space on the AS table that we have for mode-matching into the IFO. Need to figure out the desired mode and how to merge the beams best.
Attachment 1: PSLbeat.svg.png
Attachment 2: ASpath.svg.png
  13565   Sun Jan 21 13:11:25 2018 johannesUpdateDAQAcromag checks

After some research: -the- reason for the reduced +/- 20,000 swing in raw values is a default setting for having support for legacy devices enabled when using the acromag proprietary i2o peer-to-peer protocol. So this is doubly unnecessary because a) we don't have any legacy devices at all and b) we're using pure modbus/TCP and no i2o. To change the setting I have to connect via the USB configuration utility. In addition, I want to understand the averaging feature of the acromag units better, which is also configured via USB, and lets one set a fixed amount of samples to be averaged before updating the read-register value. The documentation says that the 8 channels are multiplexed into a single ADC, and that new input data is available after 10 ms for each channel, suggesting a sampling rate of 100 Hz per channel and that the multiplexing happens faster, but is not super-clear about this, so I want to test it in the cryo lab first before unleashing it onto c1auxex2.

Furthermore, the standard timing options for updating epics records are 10s, 5s, 2s, 1s, 0.5s, 0,2s, and 0.1s. On the previous c1auxex, the monitoring channels were set to 0.1s, but that clashes with the 16 Hz global EPICS rate, resulting in partial double-sampling. One can manually provide the option 0.0625s for 16Hz update rate. I will test this and how it deals with the averaging in the cryolab too.

  13576   Wed Jan 24 18:12:31 2018 johannesUpdateDAQETMX auxiliary DAQ work

I replaced the two remaining D-Sub M/M cables that still had gender-changers with M/F cables today, completing the mechanical and wiring work on the ETMX rack. All backplane adapter boards were secured to a cross-strut of the crate using zip ties. This was necessary because the adapter boards don't fit the crate with their panels attached ( the ETMX eurocrate is the only one with slightly different dimensions from all the others), and the we can't mount them to the strut using the panels. This won't be an issue on any of the other crates.



In other news:

I disabled the legacy support in the three Acromag ADC units and set the input averaging to 10 samples via the USB configuration utility. The documentation is unfortunately a little sparse about what this actually means. The manual states that "fresh input data is available to the network every 10ms", so the sampling rate is for sure faster than 100Hz. Since the IOC updates its channels every .1 seconds I assume that an averaging value of 10 to reduce the input noise is safe. The maximum value the configuration tool permits is 200. I tried this using the CryoLab DAQ and set all input channels to 200 and used StripTool to look at the time series of a slow oscillation (.1Hz) with a large amplitude (16Vpp) and looked for missed data points, indicating too long wait times for channels updates. There was no such qualitative difference between 1 sample, 10 samples, and 200 samples, so even pushing the averaging value to max seemed okay. I went with the conservative value of 10 for the ETMX DAQ, but we can likely increase this if noise on the slow inputs becomes an issue.

The input scaling of the ADC channels has been corrected. I changed the conversion method in the EPICS records from manual using the ASLO and AOFF fields to using engineering units via EGUF and EGUL. This required a little attention. The Acromags scale the dynamic input range of +/- 10V to +/- 30,000 raw value, but the EPICS IOC interprets the data type as ranging from -32767 to +32768, so the EGUF and EGUL fields must be set to -10.923 and +10.923 to achieve proper scaling. I also changed the SCAN field on all ADC channels to 0.1 seconds. This has been done for all ADC and DAC channel records.

  13578   Wed Jan 24 19:17:06 2018 johannesUpdateDAQc1auxex2 startup behavior

I compiled the burt binaries on c1auxex2 which took a little fiddling with dependencies and paths but nothing too major. The complete local epics folder (/opt/epics/) which contains the base epics binaries, modbus and burt for 32-bit linux has been copied to the shared drive at /opt/rtapps/epics-3.15.5. They belong to the most recent stable release. This was so we can now automatically call burt after the IOC initialization on c1auxex2 to restore the backed-up channel values.

I also copied the database definition and modbus instruction files to /cvs/cds/caltech/target/c1auxex2, from where they are now being read upon IOC initialization. This is an excerpt of the service file:

#ExecStart=/usr/bin/procServ -f -L /home/controls/modbusIOC/modbusIOC.log -p /run/modbusioc.pid 8008 /opt/epics/modules/modbus/bin/linux-x86/modbusApp /cvs/cds/caltech/target/c1auxex2/ETMXaux2.cmd   <-- Contains logging to file, see note 1)
ExecStart=/usr/bin/procServ -f -p /run/modbusioc.pid 8008 /opt/epics/modules/modbus/bin/linux-x86/modbusApp /cvs/cds/caltech/target/c1auxex2/ETMXaux2.cmd <-- Initializes the EPICS IOC with Modbus support
ExecStop=/bin/kill -9 ` cat /run/modbusioc.pid` <-- Kills the detached process by its process ID
ExecStartPost=/bin/bash -c "/opt/epics/extensions/bin/linux-x86/burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/latest/c1auxex.snap" <-- Restores general channel values
ExecStartPost=/bin/bash -c "/opt/epics/extensions/bin/linux-x86/burtwb -f /opt/rtcds/caltech/c1/medm/MISC/ifoalign/burt/ETMX.snap" <-- Restores PIT and YAW values from align MEDM screen
ExecStartPost=/bin/bash -c ". /home/controls/modbusIOC/ETMXaux2.sh" <-- Enables writing to PIT and YAW DAC channels, see note 2)

Note 1) I removed the logging to file for now because I noticed that if there are Acromag communication issues the logfile tends to grow in size VERY fast. In the cryo lab is had gotten to over 70GB just over the winter break. I don't think it's absolutely necessary to have it, and if diagnostics are needed we can easily uncomment it temporarily.

Note 2) I modified the static EPICS records of the four OSEM bias adjust channels so they won't start updating as soon as the IOC starts up (and before the channel defaults are restored by burt). This was done by setting the OMSL (output mode select) field from "closed_loop" to "supervisory". Sample record:

        field(DESC,"Bias Adjust for ETMX UL Coil Output")
        field(OUT, "@asynMask(C1AUXEX_XT1541A_DAC, 0, -16)MODBUS_DATA")
        field(SCAN,".1 second")
        field(OMSL,"supervisory")  <-- Used to be "closed_loop"
        field(DOL, "C1:SUS-ETMX_ULBiasSet  PP")
        field(EGU, "Volts")

Now, on reboort/IOC re-initialization the physical DAC channels are performing a one-time readback of the last stored value in the Acromag's register, then idle until the last StartPost statement executes the script ETMXaux.sh, which changes their OMSL field back to "closed_loop". This causes them to start updating their output from the calc records defined in their DOL field (which have by then recovered their default values curtesy of burt). The result is a smooth transition from idling to the controlled state with no sudden or large offset changes. yes

  13580   Wed Jan 24 23:13:30 2018 johannesUpdateDAQc1auxex2 startup behavior

The result is a smooth transition from idling to the controlled state with no sudden or large offset changes. yes

[Gautam, Johannes]

While checking how smooth the transition is we still noticed significant motion of ETMX by looking at the locked green laser and OpLevs. We found that this motion was not caused by interruption of the slow offset adjust, but rather the Watchdog being re-initialized to its OFF state, which cuts the fast channels OFF. On other optics this is observed too, but not as severe. The cause is a rather large offset on the LR coil coming from the fast DAQ, which was reported as 50mV by the slow readback channel (while other readback values are <10mV). It is present even when turning the output of the CDS model OFF, but vanishes when the watchdog is triggered. This helped us trace it to an offset of the DAC output itself: it is present at the output of the AI board but vanishes when the DAC is disconnected. The actual offset is ~40mV, as opposed to other channels on the same board, which ahve offsets in the range 3-7mV.

While we can compensate for this offset in software - it made us  wonder if the DAC channel is somehow busted and if that's what causing the 'wandering' of ETMX that we have been observing recently. There are two free DAC channels on the AI chassis that has the side coil and the green temperature control signals. We could re-route the LR signal through a different DAC channel to fix this.

gautam: 40mV offset at the AI board output gets multiplied by 3 in the dewhitening board, so there is a 120mV DC offset going to the coil (measured at dewhite board output with DMM). The offset itself isn't hurting us, but the fact that it is several times larger than other channels led us to wonder if it could be drifting around as well. From my SOS pitch balancing forays, in my head I have the number 30mrad as being the full range of the OSEM actuation - so if the offset swings by 120mV, that's ~150urad of motion, which is quite large, and is of the order of magnitude I'm used to seeing ETMX move around by.

  13590   Wed Jan 31 15:29:44 2018 johannesUpdateDAQPSL acromag server moved from megatron to c1auxex2

I moved the epics IOC server process for the single Acromag ADC that monitors the PSL signals from megatron to c1auxex2.

First, I disabled the legacy support on all channels as explained in elog 13565. Then I copied the files npro_config.cmd and NPRO.db from /opt/rtcds/caltech/c1/scripts/Acromag to /cvs/cds/caltech/target/c1psl2/ following the pattern of the old Motorola machines and the new c1auxex2. I had to make some edits for correct paths and expanded the epics records to the standard we're using for ETMX.

I then added a service to systemd on c1auxex2 that runs the epics IOC for the Acromag PSL channels: /etc/systemd/system/modbusPSL.service. No more tmux on megatron.

Running two IOCs on a signle machine at the same time did not produce any errors and seems fine so far.

  13647   Wed Feb 21 17:20:32 2018 johannesUpdateVACHornet gauge connected to DAQ.

I wired the six available BNC connectors on the front panel of the new XEND slow DAQ to physical Acromag channels. There were two unused ADC channels and eight DAC channels, of which I connected four. The following entries were added to /cvs/cds/caltech/target/c1auxex2/ETMXAUX2.db /caltech/target/c1auxex2/ETMXaux2.db

Connector Acromag Channel EPICS Name
Out2 XT1541B #5 Not Assigned
Out3 XT1541B #6 Not Assigned
Out4 XT1541B #7 Not Assigned

C1:Vac-CC1_HORNET_PRESSURE_VOLT is converted to the additional soft channel C1:Vac-CC1_HORNET_PRESSURE in units of torr using the conversion  10^{(\mathrm{Voltage}-10)} stated in the manual. A quick check showed that the resulting number and the displayed pressure on the vacuum gauge itself agree to ~1e-8 torr. Gautam added the new EPICS calc channel to the C0EDCU and restarted FB, now the data is being recorded.

Three of the output channels do not have a purpose yet, so their epics records were created but remain inactive for the time being.

Attachment 1: VacLog.png
  13681   Tue Mar 13 20:03:16 2018 johannesConfigurationComputersc1auxex replacement

I assembled the rack-mount server that will long-term replace c1auxex, so we can return the borrowed unit to Larry.


  • Intel Atom N2800 (2 cores, 1.8GHz, 1MB, 64-bit)
  • 4GB (2x2GB) DDR3 RAM
  • 128 GB SSD

IMG_20180313_105154890.jpg      IMG_20180313_133031002.jpg

I installed a standard Debian Jessie distribution, with option LXDE for minimal resource usage. Steps taken after fresh install

  1. Give controls sudo permission: usermod -aG sudo controls
  2. mkdir /cvs/cds
  3. apt-get install nfs-common
  4. Added line "chiara:/home/cds              /cvs/cds        nfs     rw,bg,nfsvers=3" to end of /etc/fstab
  5. Configured network adapter in /etc/network/interfaces
            iface eth0 inet static
            dns-search martian

    I first assigned the IP of the original c1auxex, but for some reason my ssh connections kept failing mid-session. After I switched to a different IP the disruption no longer happened.
  6. Add lines "search martian" and "nameserver" to /etc/resolv.conf
  7. apt-get install openssh-server
    At this point the unit was ready for remote connections on the martian network, and I moved it to the XEND.
  8. Added lines to /home/controls/.bashrc to set paths and environment variables:
    export PATH=/cvs/cds/rtapps/epics-$PATH
    export HOST_ARCH=linux-x86_64
    export EPICS_HOST_ARCH=linux-x86_64
    export RPN_DEFNS=~/.defns.rpn
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cvs/cds/rtapps/epics-
  9. apt-get install libmotif-common libmotif4 libxp6 (required to run burtwb utility)

The server is ready to take over for c1auxex2 and does not need any local epics compiled, since it can run the binaries in /cvs/cds.

Attachment 1: IMG_20180313_105154890.jpg
Attachment 2: IMG_20180313_133031002.jpg
  13682   Wed Mar 14 23:58:30 2018 johannesConfigurationComputersc1auxex replacement

I replaced the borrowed server with the permanent one today. Before Removing the current server, Before, I performed several additional preparations:

  • Updated Chiara hostables to IP for c1auxex
  • apt-get install procserv
  • copied ETMXaux2.* files in /cvs/cds/caltech/target/c1auxex2 to ETMXaux.* and changed references from /opt/rtcds/epics (which was a local directory on c1auxex2) to /cvs/cds/rtapps/epics- in the copied files
  • Added instruction
    to /etc/systemd/system/modbusIOC.service  (required for burtwb dependencies)

Then I replaced the server:

  1. IFO was in LSC mode with both arms locked
  2. Backed up ETMX alignment using save feature in IFOalign screen
  3. Disengaged LSC mode
  4. Shut down ETMX watchdog
  5. Disconnected ETMX satellite box
  6. Shut down c1auxex2 and c1auxex
  7. Performed the server swap
  8. Booted c1auxex
  9. Made sure EPICS channels were back online and channel defaults were restored
  10. Reconnected satellite box
  11. Turned on watchdog
  12. Turned on OpLevs
  13. Engaged LSC mode -> both arms were instantly locked

I returned c1auxex2 to Larry, who needed it back asap because of some hardware failure

Steve: Acromag XT1221 ordered 3-15-18

  13687   Mon Mar 19 14:39:09 2018 johannesConfigurationComputersc1auxex replacement

[gautam, johannes]

The temperature control output channel for the XEND seismometer wasn't working properly. The EPICS channel existed, could be written to and read from, but no physical voltage was observed on the (confirmed properly) wired connector.

The Acromag DAC that outputs this channel was completely spare in the original scheme and does not serve any other channels at the moment. We found it to be unresponsive to ping from the host machine (reminder: the Acromags are on their own subnet with IPs 192.168.114.xxx connected to the secondary ethernet adapter of c1auxex), while all others returned the ping just fine. The modules have daisy-chained ethernet connections, and the one Acromag unit behind the unresponsive one in the chain was still responding to ping and its channels were working, so it couldn't have been a problem with the (ethernet) cabling.

Gautam and I power-cycled the chassis and server, which resolved the issue. The channel is now outputting the requested voltage on the Out1 BNC connector of the chassis (front). When I was setting up the whole system and did frequent rebooting and IP-redefinitions I have seen network issues arise between server and Acromags. In particular, when changing the network settings server-side, the Acromags needed to reboot occasionally. So this whole problem was probably due to the recent server-swap, as the chassis had not been power-cycled since.


During the debugging we also found that the c1psl2 channels were not working. This was because I had overlooked to update the epics environment variables for the modbus path defined in /cvs/cds/caltech/target/c1psl2/npro_config.cmd from the local installation /opt/epics/ (which doesn't exist on the new server anymore) to the network location /cvs/cds/rtapps/epics- This has been fixed and the slow diagnostic PSL channels are recording again.

  13742   Mon Apr 9 23:28:49 2018 johannesConfigurationDAQc1psl channel list

I made a list of all the physical c1psl channels to get a better idea for how many acromags we need to replace it eventually. There  3123 unit is the one whose failure had prevented c1psl from booting, which is why it was unplugged (elog post 12852), and its channels have been inactive since. Are the 126MOPA channels used for the current mephisto? 126 tells me it's for an old lightwave laser, but I was checking a few and found that they have non-zero, changing values, so they may have been rewired.

It also hosts some virtual channels for the ISS with root C1:PSL-ISS_ defined in iss.db and dc.db, the PSL particle counter with root C1:PEM- defined in PCount.db  and a whole lot of PSL status channels defined in pslstatus.db. Transfering these virtual channels to a different machine is almost trivial, but the serial readout of the particle counter would have to find a new home.

Long story short - we need:

Function Type # Channels #Channels (no MOPA) # Units # Units (no MOPA)
ADC XT1221 34 21 5 3
DAC XT1541 17 14 3 2
BIO XT1111 19 10 2 1


3113 - ADC


3123 - ADC (failed)


4116 - DAC


XVME-210 - Binary Input


XVME-220 - Binary Output


  13764   Wed Apr 18 22:46:23 2018 johannesConfigurationGeneralAS port laser injection

Using Gautam's Finesse file and the cad files for the 40m optical setup I propagated the arm mode out of the AS port. For the location of the 3.04 mm waist I used the average distance to the ITMs, which is 11.321 m from the beam spot on the 2 inch mirror on the AS table close to the viewport. The 2inch lens focuses the IFO mode to a 82.6 μm waist at a distance of 81 cm, which is what we have to match the aux laser fiber output to.

I profiled the fiber output and obtained a waist of 289.4 μm at a distance of 93.3 cm from the front edge of the base of the fiber mount. Next step is to figure out the lens placement and how to merge the beam paths. We could use a simple mirror if we don't need AS110 and AS55, we could use a polarizing BS and work with s polarization, or we find a Faraday Isolator.

While doing a beam scan with the razor blade method I noticed that the aux laser has significant intensity noise. This is seen on the New Focus 1611 that is used for the beat signal between PSL and aux laser, as well as on the fiber output PD. There is a strong oscillation around 210 kHz. The oscillation frequency decreases when the output power is turned down, the noise eater has no effect. Koji suggested it could be light scattering back into the laser because I couldn't find a usable Faraday Isolator back when I installed the aux laser in the PSL enclosure. I'll have to investigate this a little further, look at the spectrum, etc. This intensity noise will appear as amplitude noise of the beat note, which worries me a little.

power_out_fluctuation_DC.png      power_out_fluctuation_AC_zoom.png

Attachment 1: ASpath.svg.png
  13781   Tue Apr 24 08:36:47 2018 johannesConfigurationGeneralAux Laser LD dying? (AS port laser injection)

In September 2017 I measured ~150mW output power, which was already kind of low. What are the chances of getting this one repaired? Steve, can you please check the serial number? It's probably too old like the other ones.


I suspect that the LD of the aux laser is dying.
- The max power we obtain from this laser (700mW NPRO) is 33mW. Yes, 33mW. (See attachment 1)


  13810   Thu May 3 10:40:43 2018 johannesConfigurationGeneralAS port laser injection

Instead of trying to couple the fiber output into the interferometer, I'm doing the reverse and maximize the amount of interferometer light going into the fiber. I set up the mode-matching solution shown in attachment #1 and started tweaking the lens positions. Attachment #2 shows the setup on the AS table. After the initial placement I kept moving the lenses in the green arrow directions and got more and more light into the fiber.

When I stopped this work yesterday I measured 86% of the AS port light coming out the other fiber end, and I have not yet reached a turning point with moving the lenses, so it's possible I can tickle out a little more than that.

It occured to me though that I may have been a little hasty with the placement of the mirror that in attachment #2 redirects the beam which would ordinarily go to AS55. For my arm ringdown measurements this doesn't matter, I could actually place it even before the 50/50 beamsplitter that sends light onto AS110 and double the amount of light going into the IFO. What signals are needed for the Guoy phase measurement? Is AS 110 sufficient, or do we need AS55?

Attachment 1: mm_solution_AStable.png
Attachment 2: AStable_beampath.pdf
  13832   Fri May 11 11:47:33 2018 johannesSummaryPEMAcromag issues

The replacement Acromag we scooped from the West Bridge E-Shop does actually seem to work, although we thought it was broken - at first it was just outputting zeros, but after I did the calibration procedure, applying +10 V and -10 V, respectively, it was reporting voltage correctly, over the full range. I don't know why the factory settings would be messed up, but it had been out of the box before. I did this only with channel 7, so you need to calibrate channels 0-6 and confirm that they indeed also work properly.

  13839   Sun May 13 20:48:38 2018 johannesUpdateGeneralCDS crash

I think the root of the problem is that the /opt/rtapps/ and /cvs/cds/rtapps/ mounting locations point to the same directory on the nfs server. Gautam and I were cleaning up the /cvs/cds/caltech/target/ directory, placing the previous contents of /cvs/cds/caltech/target/c1auxex/, including database files and startup instructions in /cvs/cds/caltech/target/c1auxex_oldVME/, and then moved /cvs/cds/caltech/target/c1auxex2/, which has the channel database and initialization files for the Acromac DAQ, to /cvs/cds/caltech/target/c1auxex/.

This also required updating the systemd entries on c1auxex to point to the changed directory. While confirming that everything worked as before we noticed that upon startup the EPICS IOC complains about not being able to find the caRepeater binary. This was not new and has not limited DAQ functionality in the past, but we wanted to fix this, as it seemed to be some simple PATH issue. While the paths are all correctly defined in the user login shell, systemd runs on a lower level and doesn't know about them. One thing we tried was to let systemd execute /cvs/cds/rtapps/epics- initializing EPICS. It was strange that the content of that file was pointing to /opt/rtapps/epics-, which is not mounted on the slow machines, so we changed the /opt/ it to /cvs/cds/, not realizing that the frontends read from the same directory (as Gautam said, /cvs/cds does not exist as a mount point on the frontend). It ended up not working this way, and apparently I forgot to change it back during clean up. But worse, never elogged it!

In the end, we managed to to give systemd the correct path definitions by explicitly calling them out in /cvs/cds/caltech/target/c1auxex/ETMXenv, to which a reference was added in the systemd service file. The caRepeater warning no longer appears.

ELOG V3.1.3-