I updated the figures. I think it's easier to read now.
In order to be consistent with the naming conventions for the new BLRMS filters, I made a library block that takes all the input signals of interest (i.e. for a generic optic, the coil signals, the local damping shadow sensors, and the Oplev Pitch and Yaw signals - so a total of 12 signals, unused ones can be grounded). The block is called "sus_single_BLRMS". Inside the block, I've put in 12 BLRMS library blocks, with each input signal going to one of them. All the 7 outputs of the BLRMS block are terminated (I got a compiling error if I did not do this). The idea is to identify the optic using this block, e.g. MC2_BLRMS. The BLRMS filters inside are called UL_COIL, UR_COIL etc, so the BLRMS channels will end up being called C1:SUS-MC2_BLRMS_UL_COIL_0p01_0p03 and so on. I tried implementing this in C1PEM, but immediately after compiling and restarting the model, I noticed some strange behaviour in the seismic rainbow STS strip in the control room - this was right after the model was restarted, before I attempted to make any changes to the C1PEM.txt file and add filters. I then manually opened up the filter bank screens for the RMS_STS1Z bandpass and lowpass filters, and saw that the filter switches were OFF - I wonder if this has something to do with these settings not being updated in the SDF tables? So I manually turned them on and cleared the filter hitsory for all 7 low pass and band pass filter banks, but the traces on the seismic striptool did not return to their nominal levels. I manually checked the filter shapes with Foton and they seem alright. Anyways, for now, I've reverted to the C1PEM model before I made any changes, and the seismic strip looks to be back at its normal level - when I recompiled and restarted the model with the changes I made removed, the STS1Z BLRMS bandpass and lowpass filters were ON by default again! I'm not sure what I'm doing wrong here, I will investigate this further.
To avoid the strange kicking of ETMX, I locked XARM with ITMX actuated instead of ETMX so that I changed elements of C1LSC_OUTPUT_MTRX; before: XARM=ETMX, after: XARM=ITMX.
And I change C1:LSC-XARM_GAIN from 0.007 to 0.022.
With this setup, I ran dither but then error signals of dithering oscillated as shown in the figure below.
Then I found that if C1:ASS-XARM_ETM_PIT_L_DEMOD_SIG_GAIN / C1:ASS-XARM_ETM_YAW_L_DEMOD_SIG_GAIN in C1ASS_LOCKINS_XARM.adl are changed as 0.200 -> 0.100 and 0.200 -> 0.100, respectively, the dithering works well.
But the burt file that is loaded when you let dithering "ON" is not changed, because now I don't know how to update a burt file. So, if you let dithering "ON", the dithering will run with the condition that C1:ASS-XARM_ETM_PIT_L_DEMOD_SIG_GAIN / C1:ASS-XARM_ETM_YAW_L_DEMOD_SIG_GAIN are not 0.100 but 0.200.
As I did for YARM (elog 11779), I measured the relation between offsets added just after the demodulation of the dithering loop of XARM and beam spot shift on ETMX. Defferent from YARM, the beam spot on ITMX DOES change because only BS is used as a steering mirror (TT1&2 are used for the dithering of YARM). Instead, the beam spot on BS DOES NOT change.
This time, I measured by oplevs the angles of both ETMX and ITMX for each value of offset, and using these angles I calculated the shift of the beam spot on ETMX so that I got two independent estimations (one from ETMX oplev, and the other from ITMX oplev) as shown below. The calibration of the oplevs reported in elog 11831 is taken into account.
The difference of two estimations comes from the error of calibration of oplevs and/or imperfect alignment, I think.
To focus POX beam on POX11 PD, I added an iris and a lens before POX11 PD as you can see in Attachment 1.
It seemed that the beam is well focused, but the behavior of POXDC has not changed, as shown in Attachments 2 & 3.
I had noticed for a while that the c1ioo frontend model had much higher variability than any of the other other 16k models, and would run longer than 60us multiple times an hour. This struck me as odd, since all it does is control the WFS loops. (You can see this on the Nov17 Summary page. (But somehow, the CDS tab seems broken since then, and I'm not sure why...))
This has happily now been solved! While poking around the model, I noticed that the MC2 transmission QPD data streams being sent over from c1mcs were using RFM blocks. This seemed weird to me, since I wasn't even aware that c1ioo had an RFM card. Since the c1sus and c1ioo frontends are both on the Dolphin network, I changed them to Dolphin blocks and voila! The cylce time now holds steady at 21usec.
Update: I think I figured out the problem with the CDS summary pages. Looking at the .err files in /home/40m/public_html/summary/logs on the 40m LDAS account showed that C1:FEC-33_CPU_METER wasn't found in the frame files. Indeed, this channel was commented out in c1/chans/daq/C0EDCU.ini. I enabled it and restarted daqd. Hopefully the CDS tab will come back soon...
BLRMS filters have been set up for the coil outputs and shadow sensor signals. The signals are sent to the C1PEM model from C1MCS, where I use the library block mentioned in the previous elog to put the filters in place. Some preliminary observations:
Unrelated to this work: we cleaned up the correspondence between the accelerometer numbers and channels in the C1PEM model. Also, the 3 unused ADC blocks in C1PEM (ADC0, ADC1 and ADC2) are required and cannot be removed as the ADC blocks have to be numbered sequentially and the signals needed in C1PEM come from ADC3 (as we found out when we tried recompiling the model after deleting these blocks).
Now, the beam on POX11 PD is well centered and well focused.
We found out why POXDC had behaved as reported in elog 11839. There were a few reasons: the beam was not focused enough, hight of a mirror was not matched to the beam well, path of the light reflected by misaligned SRM was occasionally close to the path of POX beam.
Then, What we did is following:
- changed orientation of SRM slightly
- changed the hight of the mirror whose hight had not matched well, by changing the pedestal (hight of which mirror was changed is shown in Attachment 1.)
- put a lens with f=250 mm (where the lens is located is shown in Attachment 1.)
- refined alignment for the POX beam to hit on the center of POX11 PD.
As a result, POX DC level behaved as shown in Attachment 2&3 when the orientation of ITMX was varied (Attachment 2: POX DC vs ITMX PIT, Attachment 3: POX DC vs ITMX YAW).
You can see broad plateau when varied in both PIT and YAW directions, and the beam is at the center of the plateau if ITMX is aligned ideally.
I glanced at the summary pages and noticed that, since Friday around when we first loaded up the new BLRMS parts, daqd has crashing very frequently (few times per hour).
I'm going to comment out the c1pem lines from the daqd master file for tonight, and see if that helps.
I added 1 line to one of the ASS scripts, UNFREEZE_DITHER.py like this:
L29> ez.cawrite('C1:ASS-'+dof+'_GAIN', 0)
The reason why I added this is: without this line, C1:ASS-'+dof+'_GAIN become larger that 1.0, which is nomial value, if you UNFREEZE DITHER when the dither is already running or C1:ASS-'+dof+'_GAIN is not 0.0.
Since removing c1pem from the daqd master file, daqd has not crashed. I suppose we're running into the stability issue that motivated us to disable some of the other models (IOPs, RFM, etc.) during the RCG upgrade.
A question to Jamie: although the new framebuilder prototype still had the same problem with trend writing, can it handle this higher testpoint/DQ channel load?
CC1 and CC2 are working again. Why did they start working again ?
The new fb1 daqd was also crashing even without the trend writing enabled. I'm not sure how much that's affected by the load, though, e.g. it might be able to handle the extra load fine but then die because of some other issue not related to the number of channels being acquired.
We should schedule some time this week to work on fb1 some more.
I changed the snapshot file for ASS, /opt/rtcds/caltech/c1/scripts/ASS_DITHER_ON.snap as follows:
L124 > C1:ASS-XARM_ETM_PIT_GAIN 1 -5.000000000000000e-02
=> C1:ASS-XARM_ETM_PIT_GAIN 1 -1.500000000000000e-02
L128> C1:ASS-XARM_ETM_YAW_GAIN 1 5.000000000000000e-02
=> C1:ASS-XARM_ETM_YAW_GAIN 1 1.500000000000000e-02
The purpose of this change is to avoid the oscillation when the dithering of X arm is running.
I've done a couple things to try and make nodus a little more secure. Some have worried that nodus may be susceptible to being drafted into a botnet, slowing down our operations.
1. I configured the ssh server settings to disallow logins as root. Ubuntu doesn't enable the root account by default anyways, but it doesn't hurt.
2. I installed fail2ban. Function: If some IP address fails to authenticate an ssh connection 3 times, it is banned from trying to connect for 10 minutes. This is mostly for thwarting mass brute force attacks. Looking at /var/log/auth.log doesn't indicate any of this kind of thing going on in the past week, at least.
3. I set up and enabled ufw (uncomplicated firewall) to only allow incoming traffic for:
I don't think there are any other ports we need open, but I could be wrong. Let me know if I broke something you need!
The noise floor of the Rga scan is glitching less today
I measured the PZT actuator gain for the Lightwave NPRO at the Y-end to be 3.6 +/- 0.3 MHz/V. This is somewhat lower than the value of 5 MHz/V reported here, but I think is consistent with that measurement.
In order to calibrate the Y-axis of my Aux PDH loop noise budget plots, I wanted a measurement of the end laser actuator gain. I proceeded to measure this as follows:
The attached plot shows the measured data. The X-axis is shown after the conversion mentioned in the last bullet point. The error bars are the standard deviations of the averaging at each DC offset.
Glitches are gone. Rga scan is good
I spent this afternoon trying to debug fb1, with very little to show for it. We're back to running from fb.
The first thing I did was to recompile EPICS from source, so that all the libraries needed by daqd were compiled for the system at hand. I compiled epics-3.14-12-2_long from source, and installed it at /opt/rtapps/epics on local disk, not on the /opt/rtapps network mount. I then recompiled daqd against that, and the framecpp, gds, etc from the LSCSoft packages. So everything has been compiled for this version of the OS. The compilation goes smoothly.
There are two things that I see while running this new daqd on fb1:
The mx stream connection between the front ends and the daqd is flaky. Everything will run fine for a while, the spontaneously one or all of the mx_stream processes on the front ends will die. It appears more likely that all mx_stream processes will die at the same time. It's unclear if this is some sort of chain reaction thing, or if something in daqd or in the network itself is causing them all to die at the same time. It is independent of whether or not we're using multiple mx "end points" (i.e. a different one for each front end and separate receiver threads in the daqd) or just a single one (all front ends connecting to a single mx receiver thread in daqd).
Frequently daqd will recover from this. The monit processes on the front ends restart the mx_stream processes and all will be recovered. However occaissionally, possibly if the mx_streams do not recover fast enough (which seems to be related to how frequently the receiver threads in daqd can clear themselves), daqd will start to choke and will start spitting out the "empty blocks" messages that are harbirnger of doom:
Aborted 2 send requests due to remote peer 00:30:48:be:11:5d (c1iscex:0) disconnected
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=005, reqn=182; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 005
mx_wait failed in rcvr eid=001, reqn=24; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 001
[Wed Dec 9 18:40:14 2015] main profiler warning: 1 empty blocks in the buffer
[Wed Dec 9 18:40:15 2015] main profiler warning: 0 empty blocks in the buffer
[Wed Dec 9 18:40:16 2015] main profiler warning: 0 empty blocks in the buffer
My suspicion is that this time of failure is tied to the mx stream failures, so we should be looking at the mx connections and network to solve this problem.
There's possibly a separate issue associated with writing the second or minute trend files to disk. With fair regularity daqd will die soon after it starts to write out the trend frames, producing the similar "empty blocks" messages.
NDS2 and the usual ports so that we can use optimus as a comsol server.
I did additional tests for the strange behavior of ASCD. ETMY, ETMX and ITMY were misaligned so that only light reflected by ITMX went into AS port. I had done similar measurement before with ITMY YAW varied.
Attachment 1 shows how ASDC level changed when ITMX PIT varied.
Attachment 2 shows how ASDC level changed when ITMX YAW varied.
Attachment 3 shows how the power of light measured by a power meter just after the AS view port varied when ITMX YAW varied.
Comparing 1 & 2, we can say that this behavior is not unique to YAW direction.
From 2 & 3, we can say something strange is happening inside the chamber.
To check if the strange behavior of ASDC is caused by SR2/SR3 or not, I did the following measurement:
ASDC measures the power of the light reflected by ITMX. POXDC measures the power of the light reflected by ITMX and SRM successively. Then I varied the angle of ITMX in YAW direction and compared the behaviors of ASDC and POXDC.
The results are shown in Attachments 1-3.
As you can see in these figures, the strange up-and-down behavior appeared ONLY in ASDC. Therefore, the cause of this behavior exists between AS table and SRM (I had confirmed that the angle of SRM did not affect ASDC).
And this behavior is fringe-like, as can be seen in the figures (there seems to be 3 "peaks" and 2 "valleys"), so the cause could be interference between main path and not good AR reflection at a mirror after SRM before AS table (I suspect a mirror is flipped mistakenly).
I took PR3 AR reflectivity and calculated PRG (PR3 is flipped and so AR surface is inside PRC).
As shown in attached figure, which shows AR specification of the LaserOptik mirror (PR3 is this mirror), AR reflectivity of PR3 is ~0.5 %. Since resonant light in PRC goes through AR surface of PR3 4 times per round trip, round trip loss due to this is ~2 %. Then I got
PRG = 7.8.
Can I ask you to make a plot of the power recycling gain as a function of the average arm loss, indicating the current loss value?
Attached is the plot of relation between the average arm round trip loss and power recycling gain. 2 % loss due to PR3 AR reflection is taken into account.
Based on calibration measurement I have done (elog 11785, 11831), I updated calibration factors of oplevs on medm screen as follows. Not to change loop gain oplev servo, I also changed oplev servo gain.
(45.1,16) => (200,3.5)
(85.6,8) => (222,3.0)
(26,-16) => (140,-3.0)
(31,-21) => (143,-4.5)
(110,8) => (122,7.2)
(81,-11) => (147,-6)
(159,15) => (239,10)
(174,-21) => (226,-16)
After the discussions at the Wednesday meeting, I redid this measurement using a sinusoidal excitation summed at the error-point of the PDH servo as opposed to a DC offset. From the data I collected, I measured the actuator gain to be 2.43 +/- 0.04 MHz/V. This is almost half the value we expect, I'm not sure if I'm missing something obvious.
Two companies are willing to make the ruby grooves and the third one is still working on their quote.
The price is ~$100 each. The cost goes down 10% if we order 50 instead of 30 pieces.
How many should we get ?
I've attached the results from my measurements of the noise characteristics of the Y-end auxiliary PDH system.
The following spectra were measured, in the range DC-1MHz:
In order to have good spectral resolution, the frequency range was divided into 5 subsections: DC-200Hz, 200Hz-3.4kHz, 3.4kHz-16.2kHz, 10kHz-100kHz, 100kHz-1MHz. The first three are measured using the SR785, while the last two ranges are measured with the Agilent network analyzer. The spectrum of the mixer output with its input terminated was quite close to the analyzer noise floor - hence, this was measured with an S560 preamplifier set to a gain of 100, and subsequently dividing the ASD by 100. To convert the Y-axis from V/rtHz to Hz/rtHz, I used two conversion factors: for the analyzer noise floor, PD dark noise, mixer noise and in-loop error signal, I made an Optickle simulation of a simple FP cavity (all parameters taken from the wiki optics page, except that I put in Yutaro's measured values for the arm loss and a modulation depth of 0.21 which I estimated as detailed here), and played around with the demodulation phase until I got an error signal that had the same qualitative shape as what I observed on an oscilloscope with the arms freely swinging (feedback to the laser PZT disabled). The number I finally used is 45.648 kHz/V (the main horns were 800mV peak-to-peak on an oscilloscope trace, results of the Optickle FP cavity simulation shown in Attachment #2 used to calibrate the X-axis). For the servo noise spectrum and in-loop control signal, I used the value of 2.43 MHz/V as determined here.
I'm not sure what to make of the strong peaks in the mixer noise spectrum between ~60Hz and 10kHz - some of the more prominent peaks are 60Hz harmonics, but there are several peaks in between as well (these have been confusing me for some time now, they were present even when I made the measurement in this frequency range using the Agilent network analyzer. My plan is to repeat these measurements for the Xend now.
Here's something to ponder.
Our online MCL feedforward uses perpendicular vertex T240 seismometer signals as input. When designing a feedforward filter, whether FIR Wiener or otherwise, we posit that the PSD of the best linear subtraction one can theoretically achieve is given by the coherence, via Psub = P(1-C).
If we have more than one witness input, but they are completely uncorrelated, then this extends to Psub = P(1-C1)(1-C2). However, in reality, there are correlations between the witnesses, which would make this an overestimate of how much noise power can be subtracted.
Now, I present the actual MCL situation. [According to Ignacio's ELOG (11584), the online performance is not far from this offline prediction]
Somehow, we are able to subtract much more noise at ~1Hz than the coherence would lead you to believe. One suspicion of mine is that the noise at 1Hz is quite nonstationary. Using median [C/P]SDs should help with this in principle, but the above was all done with medians, and using the mean is not much different.
Thinking back to one of the metrics that Eve and Koji were talking about this summer, (std(S)/mean(S), where S is the spectrogram of the signal) gives an answer of ~2.3 at that peak at 1.4Hz, which is definitely in the nonstationary regieme, but I don't have much intution into just how severe that value is.
So, what's the point of all this? We generally use coherence as a heuristic to judge whether we should bother attempting any noise subtraction in the first place, so I'm troubled by a circumstance in which there is much more subtraction to be had than coherence leads us to believe. I would like to come up with a way of predicting MISO subtraction results of nonstationary couplings more reliably.
After making sure that the upper UGFs were properly in place, I saved these settings to the SDF files. Thanks Yutaro!
To get C1PEM data back into the frames, I removed the new BLRMS blocks, recompiled, reinstalled, re-enabled it in daqd, restarted.
We still really want more headroom in our framebuilder situation.
I noticed what I thought was excessive movement of the beam spot on ITMX and ETMX on the control room monitors, and when I checked the CDS FE status overview MEDM screen, I saw that c1scx and c1asx had crashed. I ssh-ed into c1iscex and restarted both models, and then restarted fb as well. However, the DAQ-DCO_C1SCX_STATUS indicator remains red even after restarting fb (see attached screenshot). I am not sure how to fix this so I am leaving it as is for now, and the X arm looks to have settled down.
I have added a new cron job in pcdev1 at CIT using the 40m shared account. This will run the /home/40m/DetectorChar/bin/cleanarchive script one minute past midnight on the first of every month. The script removes GWsumm archive files older than 1 month old.
c1sus and c1ioo were restarted. PMC locked.
Forgot to submit this yesterday...
While we were trying to get the X-arm locked to IR using MC2, frame-builder mysteriously crashed, necessitating us having to go down to the computer and perform a hard reboot (after having closed the PSL shutter and turning all the watchdogs to "shutdown"). All the models restarted by themselves, and everything seems back to normal now..
Since there are a few hours to go before the locking efforts tonight, I've temporarily borrowed the channels used to read out the green beat frequency, and have hooked them up to the broadband IR PDs in the FOL box on the PSL table. I've used the network analyzer in the control room to roughly position the two beatnotes. I've also turned the green beat PDs back on (since the PSL shutter has to be open for the IR beat, and there is some green light falling on these PDs, but I've terminated the outputs).
So this needs to be switched back before locking efforts tonight...
With the IR beats going to the nominal ALS channels as Gautam left them, we're able to measure the free running frequency noise of the end AUX lasers.
Specifically, the end shutters are closed, leaving the AUX lasers free running. The IR beats then consist of this free running light beating with the PSL light, and the ALS phase trackers give a calibrated frequency noise spectrum. I've stabilized the PSL light by locking the laser to the Y arm via MC2 acutation, so the free running AUX laser noise should dominate by a lot above the suspension resonances. This also has the benefit of giving me the use of the CAL'd Y arm displacement as a sanity check.
At this point in time, it looks like the X laser is close to 10x noisier than the Y laser, though it does seem to be at the rule-of-thumb "10kHz/rtHz at 100Hz" level.
We were not able to fix the excess frequency noise of the AUX X laser by the usual laser diode current song and dance. Unfortunately, this level of noise is much too high to have any realistic chance of locking.
We're leaving things back in the IR beat -> phase tracker state with free running AUX lasers, on the off chance that there may be anything interesting to see in the overnight data. This may be limited by our lack of automatic beatnote frequency control. (Gautam will soon implement this via digital frequency counter). I've upped the FINE_PHASE_OUT_HZ_DQ frame rate to 16k from 2k, so we can see more of the spectrum.
For the Y beat, there is the additional weird phenomenon that the beat amplitude slowly oscillates to zero over ~10 minutes, and then back up to its maximum. This makes it hard for the phase tracker servo to stay stable... I don't have a good explanation for this.
I've set up two IPC channels that take the output from the digital frequency counters and send them to the end front-ends (via the RFM model). A summary of the steps I followed:
I've set things up such that we can select either the "PZT IN" or the frequency counter as the input to the slow servo, via means of a EPICS variable called "FC_SWITCH" (so C1:ALS-X_FC_SWITCH or C1:ALS-Y_FC_SWITCH). If this is 0, we use the default "PZT IN" signal, while setting it to 1 will change the input to the slow servo to be the frequency readout from the digital frequency counter. I've not updated the MEDM screens to reflect the two new paths yet, but will do so soon. It also remains to install appropriate filters for the servo path that takes the frequency readout as the input.
Tangentially related to this work: I've modified the FC library block so that it outputs frequency in MHz as opposed to Hz, just for convenience..
I've not updated the MEDM screens to reflect the two new paths yet, but will do so soon. It also remains to install appropriate filters for the servo path that takes the frequency readout as the input.
A few more related changes:
Here's how we should diagnose the EX laser:
A small rat / large mouse just ran through the control room. Ugh.
I'll finish up the beat / frequency noise parts of the diagnosis tomorrow later, but I've done some investigation of the AUX X laser RIN.
I placed a PDA255 at one of the rejected beams from the PBS on the downstream side of the IR faraday, making sure the power didn't saturate the PD. I measured the RIN on a SR785, and simultaneously looked at the signal on a 100MHz scope.
The RIN has a very strong dependence on the laser diode current, and no noticable dependence on the crystal temperature or the presence of the PDH modulation / temperature control cables. Here are some traces, note that "nominal" current up until recently was 2.0A.
When adjusting the diode current, a peak beings to appear in the tens of kHz, eventually noticible in the DC power trace on the scope. The point at which this occurs is not fixed.
At all times, I saw a strong intensity fluctuation at around 380-400kHz on the scope whose amplitude fluctuated a fair amount (at least 75mVrms over Vdc=6.5V, but would often be 2 or 3 times that).
I didn't look at the frequency noise while doing this, because the WiFi at the X end was too slow, I'll do more tomorrow in the daytime.
Two mechanical and two sticdky traps were set to catch univited visitor.
Absolutely no food or food remains into inside garbage cans!!!!!!!!!!!!!!!!!!!!!!!!!
When I came in this afternoon, I saw that the PZT voltage to the PMC had railed. Following the usual procedure of turning the servo gain to zero and adjusting the DC offset, I got the PMC to relock, but the PMCR level was high and the alignment looked poor on the control room monitor. So I tweaked the input alignment on the PSL till I felt it was more reasonable. The view on the control room monitor now looks more like the usual state, and the "REFL (V)" field on the PMC MEDM screen now reads 0.02-0.03 which is the range I remember it being in nominally.
I think there should be a scientifically based aveluation of the ETMY enclosure so we can make the ETMX better.
Meanwhile I'm counting pieces to move on with the south end table cover.
However, the one that was recovered from the ATF lab is no longer locking to the Rubidium reference frequency, although it was locked at the time we disconnected it from the ATF lab. I emailed the support staff at SRS, who seem to think that either the internal oscillator has drifted too far, or the Rb lamp is dead. Either ways, it needs to be repaired. They suggested that I run a check by issuing some serial commands to the unit to determine which of these is actually the problem, but I've been having some trouble setting up the serial link - I will try this again tomorrow.
The Rubidium standard we had sent in for repair and recalibration has come back. I checked the following:
However, I am still having trouble setting up a serial communications link with the FS725 with a USB-serial adaptor - I've tried with a Raspberry Pi and my Mac (using screen to try and connect), and also using one of the old Windows laptops lying around on which I was able to install the native software supplied by SRS (still using the USB-serial adaptor to establish connection though). Could it be that the unit is incompatible with the USB-serial adaptor? I had specifically indicated in the repair request that this was also a problem. In any case, this doesn't seem to be crucial, though it would have been nice for diagnostics purposes in the future...
I've stored the repaired FS725 inside the electronics cabinet (marked "Eletronics Modules") for now (the other unit was returned to Antonio in W. Bridge some weeks ago).
This LHO log indicates that EPICS slow down could be due to NFS activity. Could we make some trend of NFS activity on Chiara and then see if it correlates with EPICS flatlines?
I wonder if our EPICS issues frequency is correlated to the Chiara install.
PMC locked manually and PRM sus damping restored.