This thread: ELOG 10295
My interpretation of these ELOGs is that we did not have the replacement, and then I brought unknown fan from WB. At the same time, Steve ordered replacement fans which we found in the blue tower yesterday.
The next action is to replace the internal fan, I believe.
Steve had showed me some stock of long fibers a while back - they are from Oz Optics, and are 50m long, and are already spooled - so barring objections, we will try the MZ setup with the spooled fiber and see if there is any improvement in the fringing rate of the MZ. Then we can evaluate what additional stabilization of the fiber length is required. Anjali will upload a photo of the spooled fiber.
These weren't present last week. The peaks are present in the EX PDH error monitor signal, and so are presumably connected with the green locking system. My goal tonight was to see if the arm length control could be done using the ALS error signal as opposed to POX, but I was not successful.
Attached is my phone recording of what it sounds like right now in the PSL enclosure - not good for frequency noise measurement! The culprit is the little PC fan that is hacked onto the back of the Innolight controller.
Can we get some panel mount FC/APC connectors and put them on a box? Then we could have the whole setup inside of a box that is filled with foam and sits outside the PSL hut.
Updated the noise budget to include the unsuppressed frequency noise from the EX laser. It does not explain the noise between 10-100 Hz, although the 1-3 Hz noise is close.
Actually, I think the curve that should go on the budget is when the X arm length is locked to the PSL frequency, whereas this is when the X arm is just locally damped. I will update it later tonight.
Update 1010pm: I've uploaded the relevant plot as Attachment #2. Predictably, the unsuppressed frequency noise of the EX laser is now higher, because the MC length is a noisier frequency reference than the arm cavity. But still it is a factor of 10 below the measured ALS noise.
Next noises to budget:
I decided to use the more direct method, of disconnecting feedback to the EX laser PZT, and then looking at the cavity flashes.
Attachment #1 shows the cavity swinging through two resonances (data collected via oscilloscope). Traces are for the demodulated PDH error signal (top) and the direct photodiode signal (bottom). The traces don't look very clean - I wonder if some saturation / slew rate effects are at play, because we are operating the PD in the 30 dB setting, where the bandwidth of the PD is spec-ed as 260 kHz, whereas the dominant frequency component of the light on the PD is 430 kHz.
The asymmetric horns corresponding to the sideband resonances were also puzzling. Doing the modeling, Attachment #2, I think this is due to the fact that the demodulation phase is poorly set. The PDH modulation frequency is only ~5x the cavity linewidth, so both the real and imaginary parts of the cavity reflectivity contribute to the error signal. If this calculation is correct, we can benefit (i.e. get a larger PDH discriminant) by changing the demod phase by 60 degrees. However, for 230 kHz, it is impractical to do this by just increasing cable length between the function generator and mixer.
Anyway, assuming that we are at the phi=30 degree situation (since the measurement shows all 3 horns going through roughly the same voltage swing), the PDH discriminant is ~40 uV/Hz. In lock, I estimate that there is ~60 uW of light incident on the PDH reflection photodiode. Using the PD response of 0.2 A/W, transimpedance of 47.5 kohm, and mixer conversion loss of 6dB, the shot-noise limited sensitivity is 0.5 mHz/rtHz. The photodiode dark noise contribution is a little lower - estimated to be 0.2 mHz/rtHz. The loop does not have enough gain to reach these levels.
PDH discriminant (40 uV/Hz, see this elog)
wrong assumption - i checked the gain just now, it is G=10, and is running in the "low-noise" mode, so can only drive 4V. fixed elog, filter.
Note: While working at EX, I saw frequent saturations (red led blinking) on the SR560. Looking a the error mon signal on a scope, it had a pk-to-pk amplitude of ~200mV going into the SR560. Assuming the free-swinging cavity length changes by ~1 um at 1 Hz, the green frequency changes by ~15 MHz, which according to the PDH discriminant calibration of 40 uV/Hz should only make a 60 mV pk to pk signal. So perhaps the cavity length is changing by 4x as much, plausible during daytime with me stomping around the chamber I guess.. My point is that if the SR560 get's saturated (i.e. input > 13000 cts), the DQ-ed spectrum isn't trustworthy anymore. Should hook this up to some proper whitening electronics
G=10 or G=100?
Some time ago, I had done an actuator calibration of ITMX. This suspension hasn't been victim to the recent spate of suspension problems, so I can believe that the results of those measurement are still valid. So I decided to calibrate the in-loop error signal of the EX green PDH loop, which is recorded via an SR560, G=10, by driving a line in ITMY position (thereby modulating the X arm cavity length) while the EX green frequency was locked to the arm cavity length. Knowing the amount I'm modulating the cavity length by (500 cts amplitude sine wave at 33.14159 Hz using awggui, translating to ~17.2 kHz amplitude in green frequency), I demodulated the response in C1:ALS-X_ERR_MON_OUT_DQ channel. At this frequency of ~33 Hz, the servo gain should be large, and so the green laser frequency should track the cavity length nearly perfectly (with transfer function 1/(1+L), where L is the OLG).
The response had amplitude 5.68 +/- 0.10 cts, see Attachment #1. There was a sneaky gain of 0.86 in the filter module, which I saw no reason to keep at this strange value, and so updated to 1, correcting the demodulated response to 6.6 cts. After accounting for this adjustment, the x10 gain of the SR560, and the loop suppression, I put a "cts2Hz" filter in (Attachment #2). I had to guess a value for the OLG at 33 Hz in order to account for the in-loop suppression. So I measured the OLTF using the usual IN1/IN2 method (Attachment #3), and then used a LISO model of the electronics, along with guesses of the cavity pole (18.5 kHz), low-pass filter poles (4x real poles at 70 kHz), PZT actuator gain (1.7 MHz/V) and PDH discriminant (40 uV/Hz, see this elog) to construct a model OLTF. Then I fudged the overall gain to get the model to line up with the measurement between 1-10kHz. Per this model, I should have ~75dB of gain at ~33Hz, so the tracking error to my cavity length modulation should be ~3.05 Hz. Lines up pretty well with the measured value of 4.7 Hz considering the number of guessed parameters. The measured OLG tapers off towards low frequency probably because the increased loop suppression drives one of the measured inputs on the SR785 into the instrument noise floor.
The final calibration number is 7.1 Hz/ct, though the error on this number is large ~30%. Note that these "Hertz" are green frequency changes - so the change to the IR frequency will be half.
Attachment #4 shows the error signal in various conditions, labelled in the legend. Interpretations to follow.
I worked on characterizing the green PDH setup at EX, as part of the ALS noise budgeting process. Summary of my findings:
The main motivation was to get the residual frequency noise of the EX laser when locked to the X arm cavity - but I'll need the V/Hz PDH discriminant to convert the in-loop error signal to frequency units. The idea was to look at the PDH error signal on a scope and match up the horn-to-horn voltage with a model to back out said discriminant, but I'll have to double check my model for errors now given the large mismatch I observe in reflected power.
I tried restarting c1oaf this weekend to see if turning on the MC length FF would affect the ALS noise performance. I burtrestored the filter settings from March 2016. However, I noticed several possible anomalies, which need debugging. I am not turning the model off because of the possibility of having to reboot all the vertex FEs, but this model is totally unusable right now.
To start the noise budgeting, I decided to measure the "DFD noise", which is really the quadrature sum of the following terms:
According to past characterizations of these noises, the ADC noise level, which is expected to be at the level of a few uV/rtHz, is expected to be the dominant noise source.
The measurement was made by disconnecting the NF 1611 free space photodiode from the input to the RF amplifier on the PSL table, and connecting a Marconi (f_carrier = 40 MHz, signal level=-5dBm) instead. The phase tracked output was then monitored, and the resulting digital spectrum is the red curve in Attachment #1. The blue curve is the ASD of fluctuations of the beatnote between the PSL and EX IR beams, as monitored by the DFD system, with the X arm cavity length locked to the PSL frequency via the LSC servo, and the EX green frequency locked to the X arm cavity length by the analog PDH servo.
Assuming the Marconi frquency noise is lower than the ones being budgeted:
Since we haven't been using it, the PID control was not enabled on the doubling oven on the PSL table (it is disabled after every power outage event in the lab). I re-enabled it just now. The setpoint according to the label on the TC200 controller is 36.9 C. The PID paramaters were P=250, I=200, D=40. These are not very good as the overshoot when I turned the control on was 44 C, seems too large. The settling time is also too long, after 10 minutes, the crystal temperature as reported by the TC200 front panel is still oscillating. I can't find anything in the elog about what the nominal PID parameter values were. The X end PID seems much better behaved so I decided to try the same PID gains as is implemented there, P=250, I=60, D=25.
With the Ophir power meter, I measured 60mW of IR light going into the doubling oven, 110uW green light coming out, for a conversion efficiency of 2.7%/W, seems pretty great.
Next, I went to EX and tweaked the steering mirror alignment - I wasn't able to improve the transmission significantly using the PZT sliders on the EPICS screen, and the dither alignment servo isn't working. It required quite a substantial common mode yaw shift of the PZT mirrors to make GTRX ~ 0.5.
I plan to recover the green beat note as well and digitize it using the second available DFD channel (eventually for the Y arm) - then we can simultaneously compare the the green and IR performance (though they will have different noise floors as there is less green light on the green beat PDs, and I think lower transimpedance too).
One of the main draw backs of the measurement was the polarisation was not aligned properly in the setup. So, then the next step was to identify the polarisation at different locations in the beam path and to maximise the polarisation to either S or P component.
So, we introduced HWP at the input beam path after isolator as shown in attachment #1. Also, the polarisation was tested at positions P1, P2, P3, and P4 shown in attachment #1 by placing a polarisation beam splitter at these locations and then by observing the transmitted (P component) and reflected light (S component) using power meter.
The observations at different locations are as the follows
These observations show that the P and S components are almost equal, and this is not a good polarisation arrangement. At this point, we also had to check whether the incoming beam is linearly polarised or not.
To test the same, the PBS was placed at position P1 and the P and S components were observed with power meter as the HWP is rotated.Attachment # 2 shows the results of the same, that is the variation in P and S component as the HWP is rotated.
This result clearly shows that the input beam is linearly polarised. The HWP was then adjusted such that the P component is maximum and coupled to the MZI. With this orientation of HWP, the polarisation observed at different positions P1, P2, P3, and P4 are as follows.
This shows that the polarisation is linearly polarised as well as it is oriented along the P direction (parallel to the optical table).
We have the polarisation maintaining fiber (PM 980) as the delay fiber. The polarisation of the light as it propagates through a PM fiber depends on how well the input beam is coupled to the axis (slow or fast) of the fiber. So, the next task was to couple the light to one of the axes of the fiber.
The alignment key on the fiber is a good indication of the axis of the fiber. In our case, the alignment key lines up with the slow axis of the fiber. We decided to couple the light to the fast axis of the fiber. Since the incoming beam is P polarised, the output fiber coupler was aligned such that the fast axis is parallel to optical table as possible.
A PBS was then introduced after the fiber output collimator . There is a HWP (marked as HWP2 in attachment 1) in front of the input coupler of the fiber as well. This HWP was then rotated and observed the P and S component from the PBS that is now placed after the output coupler with a power meter.The idea was , when the light is coupled to the fast axis of the fiber, we will see the maximum at the P componet at the output
Attachment # 3 shows the observation.
In this way I tried to find the orientation of the HWP2 such that the P component is maximum at the output. But I was not succeeded in this method and observed that the output was fluctuating when the fiber was disturbed. One doubt we had was whether the fiber is PM or not . Thus we checked the fiber end with fiber microscope and confirmed that it is PM fiber.
Thus, we modifed the setup as shown in attachement # 4.The photodetector (PDA55) was monitoring the S component and the output of the detector was observed on an oscilloscope. We rotated the HWP2 such that the S component is almost minimum. At the same time, we were disturbing the fiber and was observing whether the output is fluctuating. The HWP2 angle was tweaked around the minimum of S component and observed the output with disturbing the fiber. This way we found the orientation of HWP2 such that the light is coupled to the fast axis of the fiber and the output was not fluctuating while we disturb the fiber. We tested it by heating the fiber with a heat gun as well and confirmed that the output is not fluctuating and thus the light is coupled to the fast axis of the fiber.
Is this one close to failure as well?
I set up a free-space beat on theNW side of the PSL table between the IR beam from the PSL and from EX, the latter brought to the PSL table via ~40m fiber. Initial measurements suggest very good performance, although further tests are required to be sure. Specifically, the noise below 10 Hz seems much improved.
Attachment #1 shows the optical setup.
Yehonathan came by today so I had to re-align the arms and recover POX/POY locking. This alllowed me to lock the X arm length to the PSL frequency, and lock the EX green laser to the X arm length. GTRX was ~0.36, whereas I know it can be as high as 0.5, so there is definitely room to improve the EX frequency noise suppression.
Attachment #2 shows the ALS out-of-loop noise for the PSL+X combo. The main improvements compared to this time last year are electronic.
Mix the beams in free space. We have the beam coming from EX to the PSL table, so once we mix the two beams, we can use either a fiber or free-space PD to read out the beatnote.
PSL shutter was re-opened at 6pm local time. IMC was locked. As of 10pm, the main volume pressure is already back down to the 8e-6 level.
I can't explain the mechanical switching sound Gautam reported. The relay controlling power to the TP2 forepump is housed in the main AC relay box under the arm tube, not in the Acromag chassis, so it can't be from that. I've cycled through the pumpdown sequence several times and can't reproduce the effect. The Acromag switches for TP2 still work fine.
In any case, I've made modifications to the vacuum interlocks that will help with two of the issues:
C1:AUX-PSL_ShutterRqst --> 0
After finishing this vac work, I began a new pumpdown at ~4:30pm. The pressure fell quickly and has already reached ~1e-5 torr. TP2 current and temp look fine.
However, when opening V4 (the foreline of TP1 pumped by TP2), I heard a loud repeated click track (~5Hz) from the electronics rack. Shortly after, the interlocks shut down all the TPs again, citing "AC power loss". Something is not right, I leave it to Jon and Chub to investigate.
Andrew seems to have an integrated solution of PBS+HWP in a singe mount. Or, I wonder if we should use HWP/QWP before the coupler. I am interested in a general solution for this problem in my OMC setup too.
Bob and Chub concluded that the drypump that serves as TP2's forepump had failed. Steve had told me the whereabouts of a spare Agilent IDP-7. This was meant to be a replacement for the TP3 foreline pump when it failed, but we decided to swap it in while diagnosing the failed drypump (which had 2182 hours continuous running according to the hour counter). Sure enough, the spare pump spun up and the TP2fl pressure dropped at a rate consistent with what is expected. I was then able to spin up TP1, TP2 and TP3.
Overnight pressure trends don't suggest anything went awry after the initial interlock trip. Some watchdog script that monitors vacuum pressure and closes the PSL shutter in the event of pressure exceeding some threshold needs to be implemented. Another pending task is to make sure that backup disk for c1vac actually is bootable and is a plug-and-play replacement.
The new fiber beam splitters we are ordering, PFC-64-2-50-L-P-7-2-FB-0.3W, have the slow axis working and fast axis blocked. The way the light is coupled into the fibers right now is done to maximize the amount of light into the fast axis. So we will have to do a 90deg rotation if we use that part. Probably the easiest thing to do is to put a HWP immediately before the free-space-to-fiber collimator.
Update 6pm: They have an "SB" version of the part with the slow axis blocked and fast axis enabled, same price, so I'll ask Chub to get it.
While glancing at my Vacuum striptool, I noticed that the IFO pressure is 2e-4 torr. There was an "AC power loss" reported by C1Vac about 4 hours (14:07 local time) ago. We are investigating. I closed the PSL shutter.
Jon and I investigated at the vacuum rack. The UPS was reporting a normal status ("On Line"). Everything looked normal so we attempted to bring the system back to the nominal state. But TP2 drypump was making a loud rattling noise, and the TP2 foreline pressure was not coming down at a normal rate. We wonder if the TP2 drypump has somehow been damaged - we leave it for Chub to investigate and give a more professional assessment of the situation and what the appropriate course of action is.
The PSL shutter will remain closed overning, and the main volume and annuli are valved off. We spun up TP1 and TP3 and decided to leave them on (but they have negligible load).
I renamed all channels on c1susaux2 from "C1:SUS-..." to "C1:SUS2-..." to avoid contention. When the new system is ready to install, those channel names can be reverted with a quick search-and-replace edit.
While doing this work, I noticed several errors corresponding to EPICS channel conflicts. Turns out the c1susaux2 EPICS server was left running, and the MEDM screens (and possibly several scripts) were confused. There has to be some other way of testing the new crate, on an isolated network or something - please do not leave the modbus service running as it potentially interferes with normal IFO operation. For good measure, I stopped the process and shut down the machine since I saw nothing in the elog about any running tests.
I deleted references to c1vac1 and c1vac2 (which no longer exist) and added c1vac to the autoburt request file list at /opt/rtcds/caltech/c1/burt/autoburt/requestfilelist
While Anjali is working on the 1um MZ setup, the pesky ITMY was liberated from the OSEMs. The "algorithm" :
ITMY became stuck during this process
I brought c1susaux back online this morning for suspension-channel test scripting. It had been dead for some time. I followed the procedure outlined in #12542. ITMY became stuck during this process, which Gautam tells me always happens since the last vacuum access, but ITMX is not stuck.
Attachment #1 shows the schematic of the experimental setup for the frequency noise measurement of 1 um laser source.
AUX laser will be used as the seed source and it is already coupled to a 60 m fiber (PM980). The other end of the fiber was at the AS table and we have now removed it and placed in the PSL table.
Attachment # 2 shows the photograph of the experimental setup. The orange line shows the beam that is coupled to the delayed arm of MZI and the red dotted line shows the undelayed path.
As mentioned, AUX is already coupled to the 60 m fiber and the other end of the fiber is now moved to the PSL table. This end needs to be collimated. We are planning to take the same collimator from AS table where it was coupled into before. The position where the collimator to be installed is shown in attachment #2. Also, we need to rotate the mirror (as indicated in attachment #2) to get the delayed beam along with the undelayed beam and then to combine them. As indicated in attachment #2, we can install one more photo diode to perform balanced detection.
We need to decide on which photodetector to be used. It could be NF1801 or PDA255.
We also performed the power measurement at different locations in the beam path. The different locations at which power measurement is done is shown attachment #3
There is an AOM in the beam path that coupled to the delayed arm of MZI. The output beam after AOM was coupled to the zero-order port during this measurement. That is the input voltage to the AOM was at 0 V, which essentially says that the beam after the AOM is not deflected and it is coupled to the zero-order port. The power levels measured at different locations in this condition are as follows. A)282 mW B)276 mW C)274 mW D)274 mW E)273 mW F)278 mW G)278 mW H)261 mW I)263 mW J)260 mW K)131 mW L)128 mW M)127 mW N)130 mW
It can be seen that the power is halved from J to K. This because of a neutral density filter in the path of the beam
In this case, we measured a power of 55 mW at the output of the delayed fiber. We then adjusted the input voltage to the AOM driver to 1 V such that the output of AOM is coupled to the first order port. This reduced the power level in the zero-order port of AOM that is coupled to the delayed arm of the MZI. In this case we measured a power of 0.8 mW at the output of delayed fiber.
We must be careful about the power level that is reaching the photodetector such that it should not exceed the damage threshold of the detector.
The power measured at the output of undelayed path is 0.8 mW.
We also must place the QWP and HWP in the beam path to align the polarisation.
To facilitate the 1um MZ frequency stabilization project, I decided that the AUX laser was a better candidate than any of the other 3 active NPROs in the lab as (i) it is already coupled into a ~60m long fiber, (ii) the PSL table has the most room available to set up the readout optics for the delayed/non-delayed beams and (iii) this way I can keep working on the IR ALS system in parallel. So we moved the end of the fiber from the AS table to the SE corner of the PSL table. None of the optics mode-matching the AUX beam to the interferometer were touched, and we do not anticipate disturbing the input coupling into the fiber either, so it should be possible to recover the AUX beam injection into the IFO relatively easily.
Anjali is going to post detailed photos, beam layout, and her proposed layout/MM solutions later today. The plan is to use free space components for everything except the fiber delay line, as we have these available readily. It is not necessarily the most low-noise option, but for a first pass, maybe this is sufficient and we can start building up a noise budget and identify possible improvements.
The AUX laser remians in STANDBY mode for now. HEPA was turned up while working at the PSL table, and remains on high while Anjali works on the layout.
I looked into this a little more today.
This is a problem - such large shifts in the signal level means we have to leave sufficient headroom in the choice of RF amplifier gain to prevent saturation, whereas we want to boost the signal as much as possible. Moreover, this kind of operation of tweaking the fiber seating to increase the RF signal level is not repeatable/reliable. Options as I see it:
I found the current bias output channels, C1:SUS-<OPTIC>_<DOF>BiasAdj, were all pointed at C1:SUS-<OPTIC>_ULBiasSet for every degree of freedom. This same issue appeared in all eight database files (one per optic), so it looks like a copy-and-paste error. I fixed them to all reference the correct degree of freedom.
Now the sus PD whitening bards are ready to move the back plane connectoresto the lower row and to plug the acromag interface board to the upper low.
Sus PD whitening boards on 1X5 rack (D000210-A1) had slow and fast channels mix in a single DIN96 connector. As we are going to use the rear-side backplane connector for Acromag access, we wanted to migrate the fast channel somewhere. For this purpose, the boards were modified to duplicate the fast signals to the lower DIN96 connector.
The modification was done on the back layer of the board (Attachment 1).
The 28A~32A and 28C~32C of P1 are connected to the corresponding pins of P2 (Attachment 2). The connections were thouroughly checked by a multimeter.
After the modification the boards were returned to the same place of the crate. The cables, which had been identified and noted before disconnection, were returned to the connectors.
The functionarity of the 40 (8sus*5ch) whitening switches were confimred using DTT one by one by looking at the transfer functions between SUS LSC EXC to the PD input filter IN1. All the switches showed the proper whitening in the measurments.
The PD slow mon (like C1:SUS-XXX_xxPDMon) channels were also checked and they returned to the values before the modification, except for the BS UL PD. As the fast version of the signal returned to the previous value, the monitor circuit was suspicious. Therefore the opamp of the monitor channels (LT1125) were replaced and the value came back to the previous value (attachment 3).
The parts I was waiting for arrived. I finished the beat mouth assembly, and did some characterization. Everything looks to be working as expected.
Attachment #1: Photo of the front panel. I am short of two fiber mating sleeves that are compatible with PM fibers, but those are just for monitoring, so not critical to the assembly at this stage. I'll ask Chub to procure these.
Attachment #2: Photo of the inside of the BeatMouth. I opted to use the flexible RG-316 cables for all the RF interconnects. Rana said these aren't the best option, remains to be seen if interference between cables is an issue. If so, we can replace them with RG-58. I took the opportunity to give each fiber beam splitter its own spool, and cleaned all the fiber tips.
Attachment #3: Transfer function measurement. The PDFR setup behind 1X5/1X6 was used. I set the DC current to the laser to 30.0 mA (as read off the display of the current source), which produced ~400uW of light at the fiber coupled output of the diode laser. This was injected into the "PSL" input coupler of the BeatMouth, and so gets divided down to ~100 uW by the time it reaches the PDs. From the DC monitor values (~430mV), the light hitting the PDs is actually more consistent with 60uW, which is in agreement with the insertion loss of the fiber beamsplitters, and the mating sleeves.
The two responses seem reasonably well balanced (to within 20% - do we expect this to be better?). Even though judging by the DC monitor, there was more light incident on the Y PD than on the X PD, the X response was actually stronger than the Y.
I also took the chance to do some other tests:
Attachment #4: Dark Noise analysis. I used a ZHL-500-HLN+ to boost the PD's dark noise above the AG4395's measurement noise floor. The measured noise level seems to suggest either (i) the input-referred current noise of the PD circuitry is a little lower than the spec of 16 pA/rtHz (more like 13 pA/rtHz) or (ii) the transimpedance is lower than the spec of 700 V/A (more like 600 V/A). Probably some combination of the two. Seems reasonable to me.
The optical part of the ALS detection setup is now complete. The next step is to measure the ALS noise with this sysytem. I will use the X arm for this purpose (I'd like to make the minor change of switching the existing resistive power splitter at the delay line to the newly acquired splitters which have 3dB lower insertion loss).
Thanks to new info from Johannes, I was able to finish setting up the modbus IOC on c1susaux2. It turns out the 17 Acromags draw ~1.9 A, which is way more than I had expected. Hence the reason I had suspected a short. Adding a second DC supply in parallel solves the problem. There is no issue with the wiring.
With the Acromags powered on, I carried out the following:
The modbusIOC is now running as a peristent system service, which is automatically launched on boot and relaunched after a crash. I'm able to access a random selection of channels using caget.
What's left now is to finish the Acromag-to-feedthrough wiring, then test/calibrate each channel.
Just a few remarks, since I heard from Gautam that c1susaux is next in line for upgrade.
All units have already been configured with IP addresses and settings following the scheme explained on the slow controls wiki page. I did this while powering the units in the chassis, so I'm not sure where the short is coming from. Is the power supply maybe not sourcing enough current? Powering all units at the same time takes significant current, something like >1.5 Amps if I remember correctly. These are the IPs I assigned before I left:
I used black/white twisted-pair wires for A/D, red/white for D/A, and green/white for BIO channels. I found it easiest to remove the blue terminal blocks from the Acromag units for doing the majority of the wiring, but wasn't able to finish it. I had also done the analog channel calibrations using the windows untility using multimeters and one of the precision voltage sources I had brought over from the Bridge labs, but it's probably a good idea to check it and correct if necessary. I also recommend to check that the existing wiring particularly for MC1 and MC2 is correct, as I had swapped their order in the channel assignment in the past.
While looking through the database files I noticed two glaring mistakes which I fixed:
Now that the Acromag upgrade of c1vac is complete, the next system to be upgraded will be c1susaux. We chose c1susaux because it is one of the highest-priority systems awaiting upgrade, and because Johannes has already partially assembled its Acromag replacement (see photos below). I've assessed the partially-assembled Acromag chassis and the mostly-set-up host computer and propose we do the following to complete the system.
As I go, I'm writing step-by-step documentation here so that others can follow this procedure for future systems. The goal is to create a standard procedure that can be followed for all the remaining upgrades.
The bulk of the remaining work is the wiring and testing of the rackmount chassis housing the Acromag units. This system consists of 17 units: 10 ADCs, 4 DACs, and 3 digitial I/O modules. Johannes has already created a full list of channel wiring assignments. He has installed DB37-to-breakout board feedthroughs for all the signal cable connections. It looks like about 40% of the wiring from the breakout boards to Acromag terminals is already done.
The Acromag units have to be initially configured using the Windows laptop connected by USB. Last week I wasn't immediately able to check their configuration because I couldn't power on the units. Although the DC power wiring is complete, when I connected a 24V power supply to the chassis connector and flipped on the switch, the voltage dropped to ~10V irrespective of adjusting the current limit. The 24V indicator lights on the chassis front and back illuminated dimly, but the Acromag lights did not turn on. I suspect there is a short to ground somewhere, but I didn't have time to investigate further. I'll check again this week unless someone else looks at it first.
The host computer has already been mostly configured by Johannes. So far I've only set up IP forwarding rules between the martian-facing and Acromag-facing ethernet interfaces (the Acromags are on a subnet inaccessible from the outside). This is documented in the link above. I also plan to set up local installations of modbus and EPICS, as explained below. The new EPICS command file (launches the IOC) and database files (define the channels) have already been created by Johannes. I think all that remains is to set up the IOC as a persistent system service.
Recommendation from Keith Thorne:
For CDS lab-wide, Jamie Rollins and Ryan Blair have been maintaining Debian 8 and 9 repos with some of these.
They have somewhat older EPICS versions and may not include all the modules we have for SL7.
One worry is whether they will keep up Debian 9 maintained, as Debian 10 is already out.
I would likely choose Debian 9 instead of Ubuntu 18.04.02, as not sure of Ubuntu repos for EPICS libraries.
For CDS lab-wide, Jamie Rollins and Ryan Blair have been maintaining Debian 8 and 9 repos with some of these.
They have somewhat older EPICS versions and may not include all the modules we have for SL7.
One worry is whether they will keep up Debian 9 maintained, as Debian 10 is already out.
I would likely choose Debian 9 instead of Ubuntu 18.04.02, as not sure of Ubuntu repos for EPICS libraries.
Based on this, I propose we use Debian 9 for our Acromag systems. I don't see a strong reason to switch to SL7, especially since c1vac and c1susaux are already set-up using Debian 8. Although Debian 8 is one version out of date, I think it's better to get a well-documented and tested procedure in place before we upgrade the working c1vac and c1susaux computers. When we start building the next system, let's install Debian 9 (or 10, if it's available), get it working with EPICS/modbus, then loop back to c1vac and c1susaux for the OS upgrade.
The current convention is for all machines to share a common installation which is hosted on the /cvs/cds network drive. This seems appealing because only a single central EPICS distribution needs to be maintained. However, from experience attempting this on c1vac, I'm convinced this is a bad design for the new Acromag systems.
The problem is that any network outage, even routine maintenance or brief glitches, wreaks havoc on Acromags set up this way. When the network is interrupted, the modbus executable disappears mid-execution, crashing the process and hanging the OS (I think related to the deadlocked NFS mount), so that the only way to recover is to manually power-cycle. Still worse, this can happen silently (channel values freeze), meaning that, e.g., watchdog protections might fail.
To avoid this, I'm planning to install a local EPICS distribution from source on c1susaux, just as I did for c1vac. This only takes a few minutes to do, and I will include the steps in the documented procedure. Building from source also better protects against OS-dependent buginess.
agreed - we need all pumps on UPS for their safety and also so that we can spin them down safely. Can you and Chub please find a suitable UPS?
However, I discovered that TP1---the pump that might be most damaged by a sudden power failure---is not on the UPS. It's plugged directly into a 240V outlet along the wall. This is because the current UPS doesn't have any 240V sockets. I'd recommend we get one that can handle all the turbo pumps.
Updated vac channel list is attached. There are several new ADC channels.
Hardware & Channel Assignments
All of the new hardware is now permanently installed in the vacuum rack. This includes the SuperMicro rack server (c1vac), the IOLAN serial device server, a vacuum subnet switch, and the Acromag chassis. Every valve/pump signal cable that formerly connected to the VME bus through terminal blocks has been refitted with a D-sub connector and screwed directly onto feedthroughs on the Acromag chassis.
The attached pdf contains the master list of assigned Acromag channels and their wiring.
I have updated the google doc spreadsheet to indicate the required action for the new dbfile generation.
There are three types of actions:
1. COPY - Just duplicate the old EPICS db entry. This is for soft channels, calc channels.
2. DELETE - Delete the entry for some physical channels that will not be implemented on Acromag (oplev, dewhitening mon, AI monitor, etc)
3. REPLACE - For the physical channels, we want to replace the port names.
The blue part of the spreadsheet indicates the action for each channel. If it is a physical channel, the assigned module and the channel are indicated there. What we still want to do is to use the these information for generating the port name which looks like "@asynMask(C1VAC_XT1221A_ADC 1 -16)MODBUS_DATA".
"@asynMask(C1VAC_XT1221A_ADC 1 -16)MODBUS_DATA"
The links to the spreadsheets can be found on 40m wiki: https://wiki-40m.ligo.caltech.edu/CDS/SlowControls/c1susaux
I've converted all the vac control system code to run on Python 3.4, the latest version available through the Debian package manager. Note that these codes now REQUIRE Python 3.x. We decided there was no need to preserve Python 2.x compatibility. I'm leaving the vac system returned to its nominal state ("vacuum normal + RGA").
The vac controls system is going down for migration from Python 2.7 to 3.4. Will advise when it is back up.
Work is completed and the vac system is back in its nominal state.
I'm rebooting the IOLAN server to load new serial ports. The interlocks might trip when the pressure gauge readbacks cut out.
Today I implemented protection of the vac system against extended power losses. Previously, the vac controls system (both old and new) could not communicate with the APC Smart-UPS 2200 providing backup power. This was not an issue for short glitches, but for extended outages the system had no way of knowing it was running on dwindling reserve power. An intelligent system should sense the outage and put the IFO into a controlled shutdown, before the batteries are fully drained.
What enabled this was a workaround Gautam and I found for communicating with the UPS serially. Although the UPS has a serial port, neither the connector pinout nor the low-level command protocol are released by APC. The only official way to communicate with the UPS is through their high-level PowerChute software. However, we did find "unofficial" documentation of APC's protocol. Using this information, I was able to interface the the UPS to the IOLAN serial device server. This allowed the UPS status to be queried using the same Python/TCP sockets model as all the other serial devices (gauges, pumps, etc.). I created a new service called "serial_UPS.service" to persistently run this Python process like the others. I added a new EPICS channel "C1:Vac-UPS_status" which is updated by this process.
With all this in place, I added new logic to the interlock.py code which closes all valves and stops all pumps in the event of a power failure. To be conservative, this interlock is also tripped when the communications link with the UPS is disconnected (i.e., when the power state becomes unknown). I tested the new conditions against both communication failure (by disconnecting the serial cable) and power failure (by pressing the "Test" button on the UPS front panel). This protects TP2 and TP3. However, I discovered that TP1---the pump that might be most damaged by a sudden power failure---is not on the UPS. It's plugged directly into a 240V outlet along the wall. This is because the current UPS doesn't have any 240V sockets. I'd recommend we get one that can handle all the turbo pumps.
Pin 1: RxD
Pin 2: TxD
Pin 5: GND
Baud rate: 2400
Data bits: 8
Stop bits: 1
I'm running a test to see how stable the EX green lock is. For this purpose, I've left the slow temperature tuning servo on (there is a 100 count limiter enabled, so nothing crazy should happen).