As before, I am unable to get data from the past. With DTT on Allegra I got data from now, but its unavailable from 1 hour ago. Same problem using mDV on mafalda. I blame Joe again - or the military industrial complex.
Although trends are available, I am unable to get any full data from in the past (using DTT or DV). I started the FB's daqd process a few times, but no luck.
I blame Joe's SimPlant monkeying from earlier today for lack of a better candidate. I checked and the frames are actually on the FB disk, so its something else.
I tried running dataviewer and dtt this morning. Dataviewer seemed to be working. I was able to get trends, full data on a 2k channel (seismic channels) and full data on a 16k channel (C1:PEM-AUDIO_MIC1) This was tried for a period 24 hours a go for a 10 minute stretch.
I also tried dtt and was able to get 2k and 16k channel data, for example C1:PEM-AUDIO_MIC1. Was this problem fixed by someone last night or did time somehow fix it?
DAQ reload/restart was performed at about 1315 PST today. The previous ini file was backed up as c1pem20120309.ini in the /chans/daq/working_backups/ directory.
I set the following to record:
The two JIMS channels at 2048:
[C1:PEM-JIMS_CH1_DQ] Persistent version of JIMS channel. When bit drops to zero indicating something bad (BLRMS threshold exceeded) happens the bit stays at zero for >= the value of the persist EPICS variable.
[C1:PEM-JIMS_CH2_DQ] Non-persistent version of JIMS channel.
And all of the BLRMS channels at 256:
Names are of the form:
On monday I intend to look at the weekend seismic data to establish thresholds on the JIMS channels.
256 was the lowest rate possible according to the RCG manual. The JIMS channels are recorded at 2048 because I couldn't figure out how to disable the decimation filter. I will look into this further.
Yesterday, when I worked on the damping servo, I found that any of the daqvtools (ndscope, dtt, dataviewer,...) is not available. We may need to restart the fb and rt machines.
Here is another trick for the DAQ setup when you add a DAQ channel associated with a new front end code.
Once you finish setting up the things properly according to this wiki page (this page ), you have to go to
and then edit the file called master.
This file contains necessary path where fb should look at, for the daqd initialization.
Add your path associated with your new front end code on this file, for example:
After editing the file, restart the daqd on fb by the usual commands:
telnet fb 8088
I investigated this issue today. At first, it seemed that only new suspension testpoints are inaccessible. I was able to use diaggui for a measurement on MC2. The DAQ network cable between 1X4 and 1Y1 was tied and is very taught now (we should relieve this as soon as possible, best solution is to lay down a longer cable over the bridge). My hypothesis is that the DAQ network might have broken while tying this cable and it probably did not come back since then.
The simplest solution would have been to restart c1su2 models. As I restarted those models though, I found that c1lsc and c1sus models failed. This is very unusual as c1su2 models are independent and share nothing with the other vertex models. So I had to restart all the FE computers eventually. But this did not solve the issue. Worse, now the DAQ isn't working for the vertex machiens as well.
Next step was to try restarting fb1 itself. We switched off all the FE computers, restarted fb1, stopped daqd_* processes, reloaded gpstime module, restarted open-mx, mx, nds and daqd_* process. But the mx.service failed to load with following error message:
● mx.service - LSB: starts MX driver for Myrinet card
Loaded: loaded (/etc/init.d/mx)
Active: failed (Result: exit-code) since Mon 2022-04-25 17:18:02 PDT; 1s ago
Process: 4261 ExecStart=/etc/init.d/mx start (code=exited, status=1/FAILURE)
Apr 25 17:18:02 fb1 mx: Loading mx driver
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_mcp.ko: File exists
Apr 25 17:18:02 fb1 mx: insmod: ERROR: could not insert module /opt/mx/sbin/mx_driver.ko: File exists
Apr 25 17:18:02 fb1 systemd: mx.service: control process exited, code=exited status=1
Apr 25 17:18:02 fb1 systemd: Failed to start LSB: starts MX driver for Myrinet card.
Apr 25 17:18:02 fb1 systemd: Unit mx.service entered failed state.
(Ignore the timestamp above, I ran the command again to capture the error message.)
However, I was able to start all the FE models without any errors and daqd processes are also all running without showing any errors. Everything is green in CDS screen with no error messages. But the only thing still wrong is mx.service which is not running.
From my limited knowledge and experience, mx.service is a one-time script that mounts mx devices in /dev and loads the mx driver. I tried running the script /opt/mx/sbin/mx_start_stop :
This gave the same error. On searching little bit online, "insmod: ERROR; cound not insert module" error comes up when the kernel version of the driver doesnot match the Linux kernel (whatever that means!). Such deep issues should not appear out of nowhere in a previosuly perfectly runnig system. I'll check around more what changed in fb1, network cables etc.
Its pretty exciting to see that Joe got Alex to actually use the ELOG. Its a proof that even rare events occur if you are patient enough.
1) I fixed the MEDM links to point to the new sitemap.adl in /opt/rtcds. There is a link on the new sitemap which points to the old sitemap so that there is nothing destroyed yet.
2) Some of the fields in the screen are white. These are from the new c1sus processor, not issues with the slow controls. I think its just stuff that has not yet been created in the C1SUS simulink module.
3) The PZT steering controls are gone. Without this we cannot get the beam down the arm. Must fix before aliging things after the MC. Since PZT used to be controlled by ASC, we'll have to wire the Piezo Jena PZT controls in from a different VME 4116. Possibly c1iool0's crate?
4) Also, the IPANG and IPPOS are somehow not working right. I guess this is because they are part of the ASC / the old ETMX system. We'll have to wire the IPANG QPD into the new ETMY ADC system if we want to get the initial alignment into the Y-arm correct.
5) I've started migrating things over from the old SITEMAP. Please just use the new SITEMAP. it has a red link to the old one, but eventually everything on the new one will work after Joe, Alex, me, and Kiwamu are done tweaking.
The frame builder is timed from the Symmetricom GPS card now, which is getting the IRIGB timecode from the freq. distribution amplifier (from the VME GPS receiver card).
I have adjusted the GPS seconds to match the real GPS time and the DTT seems to be happy: sweeping MC2 MCL filter module produces nice plot.
Test points are working on SUS.
Excitations are working on SUS.
I am leaving the frame builder running and acquiring the data.
Since we now have a good measurement of the phase noise of the Rb clock Marconi locked to the Rb clock, I wanted to use that to check out the old DAQ system:
I used Megan's phase noise setup - Marconi #2 is putting out 11000013 Hz at 13 dBm into the ZP-3MH mixer. Marconi #1 is putting out 3 dBm at 11000000 Hz into the RF input.
The output goes through a 50 Ohm load and then a Mini-Circuits BNC LP filter (either 2 or 5 MHz). Then an SR560 set for low noise, G = 5, AC coupling, 1-pole LP @ 1 kHz.
This SR560 output goes into the channel C1:IOO-MC_DRUM1 (which is sampled at 16384 Hz with ICS-110B after the usual Sander Liu AA chassis containing the INA134s).
This is the 0.3 mHz BW spectrum of this test - as you can see the apparent linewidth (assuming the width is all caused by the DAQ jitter) is comparable to the BW and therefore not resolved.
Basically, the Hanning window function is not sharp enough to do this test and so I will do it offline in Matlab.
I heard a rumor about a DAQ problem at the 40m.
To investigate, I tried retrieving data from some channels under C1:SUS-AS1 on the c1sus2 front end. DQ channels worked fine, testpoint channels did not. This pointed to an issue involving the communication with awgtpman. However, AWG excitations did work. So the issue seemed to be specific to the communication between daqd and awgtpman.
daqd logs were complaining of an error in the tpRequest function: error code -3/couldn't create test point handle. (Confusingly, part of the error message was buffered somewhere, and would only print after a subsequent connection to daqd was made.) This message signifies some kind of failure in setting up the RPC connection to awgtpman. A further error string is available from the system to explain the cause of the failure, but daqd does not provide it. So we have to guess...
One of the reasons an RPC connection can fail is if the server name cannot be resolved. Indeed, address lookup for c1sus2 from fb1 was broken:
$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)
$ host c1sus2
Host c1sus2 not found: 3(NXDOMAIN)
In /etc/resolv.conf on fb1 there was the following line:
Changing this to search martian got address lookup on fb1 working:
$ host c1sus2
c1sus2.martian has address 192.168.113.87
$ host c1sus2
c1sus2.martian has address 192.168.113.87
But testpoints still could not be retrieved from c1sus2, even after a daqd restart.
In /etc/hosts on fb1 I found the following:
Changing the hardcoded address to the value returned by the nameserver (192.168.113.87) fixed the problem.
It might be even better to remove the hardcoded addresses of front ends from the hosts file, letting DNS function as the sole source of truth. But a full system restart should be performed after such a change, to ensure nothing else is broken by it. I leave that for another time.
[Anchal, Paco, JC]
Thanks Chris for the fix. We are able to access the testpoints now but we started facing another issue this morning, not sure how it is related to what you did.
The steps we have tried to fix this are:
These above steps did not fix the issue. Since we have the testpoints (C1:SUS-ETMX_TRX_OUT & C1:SUS-ETMY_TRY_OUT) for now to monitor the transmission levels, we are going ahead with our upgrade work without resovling this issue. Please let us know if you have any insights.
It looks like the RFM problem started a little after 2am on Saturday morning (attachment 1). It’s subsequent to what I did, but during a time of no apparent activity, either by me or others.
The pattern of errors on c1rfm (attachment 2) looks very much like this one previously reported by Gautam (errors on all IRFM0 ipcs). Maybe the fix described in Koji’s followup will work again (involving hard reboots).
As a test, I did a remote reboot of both Megatron and c1iscex, to make sure there was no code running that might interfere with the dataviewer. Megatron is behind a firewall, so I don't see how it could be interfering with the frame builder. c1iscex was only running a test module from earlier today when I was testing the multi-filter matrix part. No daqd or similar processes were running on this machine either, but it is not behind a firewall at the moment.
Neither of these seemed to affect the lack of past data. I note the error message from dataviewer was "read(); errno=9".
Going to the frame builder machine, I ran dmesg. I get some disturbing messages from May 26th and June 7th. There are 6-7 of these pairs of lines for each of these days, spread over the course of about 30 minutes.
Jun 7 14:05:09 fb ufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Jun 7 14:11:14 fb last message repeated 19 times
There's also one:
Jun 7 13:35:14 fb syslogd: /usr/controls/main_daqd.log: No space left on device
I went to /usr/controls/ and looked at the file. I couldn't read it with less, it errored with Value too large for defined data type. Turns out the file was 2.3 G. And had not been updated since June 7th. There were also a bunch of core dump files from May 25th, and a few more recent. However the ones from May 25th were somewhat large, half a gig each or so. I decided to delete the main_daqd.log file as well as the core files.
This seems to have fixed the data history for the moment (at least with one 16k channel I tested quickly). However, I'm now investigating why that log file seems to have filled up, and see if we can prevent this in the future.
Jamie, I think the computers know that you are away. c1lsc keeps going down.
The short time plots are correct.
Is there some indication from the attached image that there is a problem with c1lsc? I see some drop outs in the channels you're plotting, but those are not c1lsc channels.
The channels with the drop outs are I think derived channels, as opposed to ones that are generated on the front end. Therefore they could have been affected by the c1auxey outages from earlier in the week.
Today Alex came over, performed his magic rituals on the DAQAWG computer and fixed it. Now it's up and running again.
I asked him what he did, but he's not sure of what fixed it. He couldn't remember exactly but he said that he poked around, did something somewhere somehow, maybe he tinkered with tpman and eventually the computer went up again.
Now everything is fine.
Yet again, the DAQAWG flipped out for an unknowable reason. In order of restart activities listed on the Wiki, I keyed the crate and nothing really happened, then I hit the physical reset button and nothing happened, and then I did the 'telnet....vmeBusReset', and a couple minutes later, it was all good again.
I am re-starting work on the daqd upgrade again now. Expect the daqd to be offline for most of the day. I will report progress.
Summary: new daqd code running overnight test on fb1. Stability issues persist.
The code is from Keith's "tests/advLigoRTS-40m" branch, which is a branch of the current trunk. It's supposed to include patches to fix the crashes when multiple frame types are written to disk at the same time. However, the issue is not fixed:
2016-06-07_20:38:55 about to write frame @ 1149392336
2016-06-07_20:38:55 Begin Full WriteFrame()
2016-06-07_20:38:57 full frame write done in 2seconds
2016-06-07_20:39:11 about to write frame @ 1149392352
2016-06-07_20:39:11 Begin Full WriteFrame()
2016-06-07_20:39:13 full frame write done in 2seconds
2016-06-07_20:39:27 about to write frame @ 1149392368
2016-06-07_20:39:27 Begin Full WriteFrame()
2016-06-07_20:39:29 full frame write done in 2seconds
2016-06-07_20:39:43 about to write second trend frame @ 1149391800
2016-06-07_20:39:43 Begin second trend WriteFrame()
2016-06-07_20:39:43 about to write frame @ 1149392384
2016-06-07_20:39:43 Begin Full WriteFrame()
2016-06-07_20:39:44 full frame write done in 1seconds
2016-06-07_20:39:59 about to write frame @ 1149392400
2016-06-07_20:40:04 Begin Full WriteFrame()
2016-06-07_20:40:04 Second trend frame write done in 21 seconds
2016-06-07_20:40:14 [Tue Jun 7 20:40:14 2016] main profiler warning: 1 empty blocks in the buffer
2016-06-07_20:40:15 [Tue Jun 7 20:40:15 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:16 [Tue Jun 7 20:40:16 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:17 [Tue Jun 7 20:40:17 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:18 [Tue Jun 7 20:40:18 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:19 [Tue Jun 7 20:40:19 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:20 [Tue Jun 7 20:40:20 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:21 [Tue Jun 7 20:40:21 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:22 [Tue Jun 7 20:40:22 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:23 [Tue Jun 7 20:40:23 2016] main profiler warning: 0 empty blocks in the buffer
This failure comes when a full frame (1149392384+16) is written to disk at the same time as a second trend (1149391800+600). It seems like every time this happens daqd crashes.
I have seen other stability issues as well, maybe caused by mx flakiness, or some sort of GPS time synchronization issue caused by our lack of IRIG-B cards. I'm going to look to see if I can get the GPS issue taken care of so we take that out of the picture.
For the last couple of hours I've only seen issues with the frame writing every 20 minutes, when the full and second trend frames happen to be written at the same time. Running overnight to gather more statistics.
38 restarts overnight. Problem definitely not fixed. I'll be reverting back to old daqd and fb this morning. Then regroup and evaluate options.
Q has pointed out that we expect a sign flip in the AS55 signal for DARM as we reduce the CARM offset in the PRFPMI case. Koji also mentioned that the SRC will help broaden the DARM linewidth. I wanted to check and think about these things with my Optickle simulation. Q is working on confirming my results with Mist.
The simulation situation:
* The demod phase for AS55 is set in the MICH-only case so that the MICH signal is maximized in the Q-phase. I do not change the demod phase at all in these simulations.
* Cavity lengths (arms, recycling cavities) are the measured lengths.
* I look at AS55 I and Q as DARM sensors (i.e. I'm doing DARM sweeps) as a function of CARM offset, for both PRFPMI and DRFPMI cases.
Spoiler alert! Conclusions:
* In the PRFPMI case, the DARM signal shows up with approximately equal strength in the I and Q phases, so we suffer only about a factor of 2 if we do not re-optimize the demod angle for AS55.
* In the DRFPMI case, the DARM signal is a factor of 1,000 smaller in the Q-phase than the I-phase, which means that the ideal demod phase angle has moved by about 90 degrees from the MICH-only case. We must either use the I-phase signal or change the demod phase by 90 degrees in order to acquire lock.
* In the PRFPMI case, there is a sign flip for DARM on the AS55 PD around 100pm, so we don't want to use AS55 for DARM until we are well inside 50pm, and aren't going to fluctuate out of that range.
* In the DRFPMI case, there is no such sign flip, at least out to 1nm, so we can use AS55 for DARM as soon as we see a viable signal. This is super awesome. The caveat is that the gain changes significantly as we reduce the CARM offset, so we either need a UGF servo (eventually) or careful watching (for now).
* The AS55 linear(-ish) range is much broader in the dual recycled case, which is yet another reason why getting DARM on AS55 will be easier for DRFPMI.
Why didn't we do it already?
* To put the SRM QPD back, we'd also have to move Steve/EricG's laser. Since I had other things to do, I left the setup for tonight, but I think I will want it for tomorrow night.
* Monday night (and tonight) we can pretty reliably get DARM onto AS55Q for the PRFPMI case, and I don't know what the cause has been for my locklosses, so I thought I'd try to figure that out first.
First up, the current transition we've been trying to handle, PRFPMI DARM to AS55Q. I also plot AS55I, and we see that the signals are roughly the same magnitude (the x axis isn't the same between these plots! sorry), so we aren't screwed if we don't change the demod phase angle. We'll be better off once we can do a re-optimization, but this is assuming we are stuck (at first) with our MICH-only demod phase angle.
Next up, the same plots, but for the DRFPMI case. Note here that there is a factor of about 1000 in the y-axis scales, and also that there is no switch in the sign of the zero-crossing slope for the I-phase.
And here is the same data (DRFPMI case), but zoomed out for the Q-phase, so you can see the craziness of this phase. Again, this is much smaller than the signals in the I-phase, so I'm not too worried.
* Steve leaves the SRM oplev back in its nominal location (we can worry about aligning the mirror, and aligning the beam on the PD, but please put it back approximately where it came from).
* Try DRMI + 2 arm locking, which I don't think we have ever actually done before. Hopefully there won't be any tricks, and we can get to an equivalent place and successfully get DARM to AS55.
* .... Keep going?
Note the effect of quadrature rotation for small offsets.
We'd like to know how much actuation is required on the ETMs to lock the DARM degree of freedom. The "disturbance" we are trying to cancel is the seismic driven length fluctuation of the arm cavity. In order to try and estimate what the actuation required will be, we can use data from POX/POY locks. I'd collected some data on Friday which I looked at today. Here are the results.
If this approach looks legit, I will compute the control signal that is required to stabilize this level of disturbance using the DARM control loop, and see what is the maximum permissible series resistance we can use in order to realize this stabilization. We can then compare various scenarios like different whitening schemes, with/without Barry puck etc, and look at coil driver noise levels for each of them.
Here is an updated plot - the main difference is that I have added a trace that is the frequency domain wiener filter subtraction of the coherent power between the L_X and L_Y time series. I tried reproducing the calculation with the time domain wiener filter subtraction as well, using half of the time series (i.e. 5 mins) to train the wiener filter (with L_X as target and L_Y as witness), but I don't get any subtraction above 5 Hz on the half of the data that is a test data set. Probably I am not doing the pre-filtering correctly - I downsampled the signal to 1 kHz, weighted it by low passing the signal above 40 Hz and trained the Wiener filter on the resulting time series. But this frequency domain Wiener filter subtraction should be at least a lower bound on DARM - indeed, it is slightly lower everywhere than simply taking the time domain subtraction of the two data streams.
Putting a slightly cleaned up version of this plot in now - I'm only including the coherence-inferred DARM estimate now instead of the straight up time domain subtraction. So this is likely to be an underestimate. At low (<10 Hz) frequencies, the time domain computation lines up fairly well, but I suspect that I am getting huge amounts of spectral leakage (see Attachment #2) in the way I compute the spectrum using scipy's filtering routine (once the Wiener filter has been computed). Note that Attachment #2, I didn't break up the data into a training/testing set as in this case, we just care about the one-off offline performance in order to get an estimate of DARM.
The python version of the wiener filter generating code only supports [b,a] output of the digital filter, an sos filter might give better results. Need to figure out the least painful way of implementing the low-noise digital filtering in python...
Using the Wiener Filter estimate of the DARM disturbance we will have to cancel, I computed how the control signal would look like for a few scenarios. Our DACs are 16-bit, +/-10V (i.e. +/-32,768cts-pk, or ~23,000cts RMS). We need to consider the shape of the de-whitening filter to conclude whether it is feasible to increase the series resistance by x10 or not.
Note that in this first computation, I have not considered
While doing this calculation, I have accounted for the fact that right now, the analog de-whitening filters in the ETM drive chain have a x3 gain which we will remove. Actually this is an assumption, I have not yet measured a transfer function, maybe I'll do one channel at EY to confirm. Also, the actuator gains themselves need to be confirmed.
As I was looking at the coil driver schematic more closely, I realized that there are actually two separate series resistances, one for the fast controls path, and another for the DC bias voltage from the slow ADCs. So I think we have been underestimating the Johnson noise of the coil drivers by sqrt(2). I've also attached screenshots of the IFOalign and MCalign screens. The two ITMs and ETMX have pitch DC bias values that are compatible with a x10 increase of the series resistance. But even so, we will have ~3pA/rtHz per coil from the two resistances.
gautam 8pm May8: Seems like I had confirmed the x3 gain in the EX de-whitening board when Johannes and I were investigating the AI board offset.
I've been able to repeatedly get off of ALS and onto (TRY-TRX)/(TRY+TRX). Nevertheless, lock is lost between arm powers of 10 and 20.
I do the transition at the same place as the CARM->SqrtInv transition, i.e. arm powers about 1.0 Jenne started a script for the transition, and I've modified it with settings that I found to work, and integrated it into the carm_cm_up script. I've also modified carm_cm_down to zero the DARM normalization elements.
I was thwarted repeatedly by the frequent crashing of daqd, so I was not able to take OLTFs of CARM or DARM, which would've been nice. As it was, I tuned the DARM gain by looking for gain peaking in the error signal spectrum. I also couldn't really get a good look at the lock loss events. Once the FB is behaving properly, we can learn more.
Turning over to difference in transmission as an error signal naturally squashes the difference in arm transmissions:
I was able to grab spectra of the error and control signals, though I did not take the time to calibrate them... We can see the high frequency sensing noise for the transmission derived signals fall as the arm power increases. The low frequency mirror motion stays about the same.
So, it seems that DARM was not the main culprit in breaking lock, but it is still gratifying to get off of ALS completely, given its out-of-loop-noise's strong dependence on PSL-alignment.
We have done several things this evening, which have incrementally helped the lock stability. We are still locking CARM and DARM on ALS, and PRMI on REFL165.
Something that has been bothering me the last few days is that early in the evening, I would be able to get to very high arm powers, but later on I couldn't. I think this has to do with setting the contrast at the AS port separately for the sideband versus the carrier. I had been minimizing the AS port power with the arms held off resonance, PRMI locked. But, this is mostly sideband. If instead I optimize the Michelson fringes when the arms are held with ALS at arm powers of 1, and PRM is still misaligned, I end up with much higher arm powers later. Some notes about this though: most of this alignment was done with the arm cavity mirrors, specifically the ETMs, to get the nice Michelson fringes. When the PRM is restored and the PRMI locked, the AS port contrast doesn't look very good. However, when I leave the alignment alone at this point, I get up to arm powers above 100, whereas if I touch the BS, I have trouble getting above 50.
Around GPS time 1101094920, I moved the DARM offset after optimizing the CARM offset. We were able to see a pretty nice zero crossing in AS55, although that wasn't at the same place as the ALS diff zero offset (close though). At this time, the arm powers got above 250, and TRY claimed almost 200. These are the plots below, first as a wide-view, then zoomed in. During this time, PRCL still has a broadband increase in noise when the arm powers are high, and CARM is seeing a resonance at a few tens of Hz. But, we can nicely see the zerocrossing in AS55, so I think there's hope of being able to transition DARM over.
Now, the same data, but zoomed in more.
During the 40m meeting, we had a few ideas of directions to pursue for locking:
In order to estimate the free-running DARM displacement noise, I measured the DARM OLTF using the usual IN1/IN2 prescription. The measured data was then used to fit some model paramters for a loop model that can be used over a larger frequency range.
In summary, the UGF is ~150 Hz and phase margin is ~30 deg. This loop would probably benefit from some low-pass filter being turned on.
As Jenne mentioned, I created a model of the DARM OLG to see why we have so little phase margin. However, it turns out I can explain the phase after all.
Chris sent me his work for the aLIGO DARM phase budget, which I adapted for our situation. Here's a stacked-area plot that shows the contributions of various filters and delays on our phase margin, and a real measurement from a few days ago .
This isn't so great! Informed by Chris's model, the digital delays look like: (Here I'm only listing pure delays, not phase lags from filters)
This adds up to about 570usec, 20.5 degrees at 100Hz, largely due to the sheer number of computer hops the transmission loops involve.
As a check, I divided the measured OLG by my model OLG, to see if there is any shape to the residual, that my model doesn't explain. It looks like it fits pretty well. Plot:
So, unless we undertake a bunch of computer work, we can only improve our transmission loops through our control filter design.
Everything I used to generate these plots is attached.
I think the dolphin and RFM transit times are double-counted in this budget. As I understand it, all IPC transit times are already built in to the cycle time of the sending model. That is, the sending model is required to finish its computational work a little bit early, so there's time left to transmit data to the receivers before the start of the next cycle. Otherwise you get IPC errors. (This is why the LSC models at the sites can't use the last ~20 usec of their cycle without triggering IPC errors. They have to allow that much time for the RFM to get their control signals down the arms to the end stations.)
For instance, the delay measurement in elog 9881 (c1als to c1lsc via dolphin) shows only the c1lsc model's own 61 usec delay. If the dolphin transfer really took an additional cycle, you would expect 122 usec.
And in elog 10811 (c1scx to c1rfm to c1ass), the delay is 122 usec, not because the RFM itself adds delay, but because an extra model is traversed.
Bottom line: there may still be some DARM phase unaccounted for. And it would definitely help to bypass the c1rfm model, as suggested in 9881.
Tonight we noticed that the REFL_DC signal has gone bipolar, even though the whitening gain is 0 dB and the whitening filter is requested to be OFF.
Fixed! I noticed that whitening gain changes weren't having any effect on CM_SLOW. I then checked REFL_DC, where this also seemed to be the case. Since the gain is controlled via VME machine, and whitening filter switching is controlled via RCG, I figured there must be something wrong with the board. I checked all of the DC PD signals, which share a whitening filter board, and they all had the same symptoms.
I went and peeked at the board, and it turns out the backplane cable had fallen off.
I plugged it in, things look ok.
I have added another block to the LSC screen (and made the corresponding sub-screen) to expose the analog settings for the DC photodiodes.
Note that we have 2 open channels there, which are still called something like "PD2" and "PD3" from olden times.
If we ever chose to use those, we will probably want to change their names, in /cvs/cds/caltech/target/c1iscaux2/LSC_aux2.db and /cvs/cds/caltech/target/c1iscaux/LSC_aux.db
I installed a DC PD (Thorlabs PDA 520) in the beam path to AS55. I placed a 2" 90/10 BS on a flip mount that picks of enough light for the PD to spit out ~8V when the port is bright. Single arm continuous signal will be ~2V. While most of the light still continues towards AS55, the displacement from the BS moves the beam off AS55, so I used the flip mount in case anyone needs to use AS55. The current configuration is UP.
When we're done with loss investigations the flip mount should be removed from the bench.
I hooked the PD up to an ethernet-enabled scope and started scripting the loss map measurement (scope can receive commands via http so we can automate the data acquisition). The scope that was present at the bench and had been used for the MC ringdown measurements had a 'scrambled' screen that I couldn't fix so I had to retrieve another scope ("scope1"). I'll try to find out what's wrong with it but we may have to send it in for repair.
DC Power Strip Assemblies delivered and stored behind the Y arm tube (Attachment 1)
I also moved the spare 1U Chassis to the same place.
Lock acquisition working well tonight. Was able to engage CM boost (not superboost) with bandwidth of ~10kHz. Also succeeded once in handing off DARM to DC readout.
These were delivered to the 40m today and are on Rana's desk
I'll order a couple of these (5 ordered for delivery on Wednesday) in case there's a hot demand for the jack / plug combo that this one has.
Rana and I talked about some (genius) options for the large range DC bias actuation on the SOS, which do not require us to supply high-voltage to the OSEMs from outside the vacuum.
What we came up with (these are pretty vague ideas at the moment):
For the thermal option, I remembered that (exactly a year ago to the day!) when we were doing cavity mode scans, once the heaters were turned on, I needed to apply significant correction to the DC bias voltage to bring the cavity alignment back to normal. The mechanism of this wasn't exactly clear to me - furthermore, we don't have a FLIRcam picture of where the heater radiation patter was centered prior to my re-centering of it on the optic earlier this year, so we don't know what exactly we were heating. Nevertheless, I decided to look at the trend data from that night's work - see Attachment #1. This is a minute trend of some ETMY channels from 0000 UTC on 18 July 2018, for 24 hours. Some remarks:
Also see this elog by Terra.
Attachment #2 shows the results from today's heating. I did 4 steps, which are obvious in the data - I=0.6A, I=0.76A, I=0.9A, and I=1.05A.
In science, one usually tries to implement some kind of interpretation. so as to translate the natural world into meaning.
The IFO is more or less back to an operational state. Some details:
One error persists - the "DC" indicator (data concentrator?) on the CDS medm screen for the various models spontaneously go red and return to green often. Is this a known issue with an easy fix?
I think I fixed the DC error issue
1. I added the leap second (leapsecond ?) entry for 2016/12/31, 23:60:00 UTC to daqdrc
set gps_leaps = 820108813 914803214 1119744016;
set gps_leaps = 820108813 914803214 1119744016 1167264018;
2. Restarted FB and all realtime models
Now I don't see any RED light.
I installed a DC power strip (24 V variant, 12 outlets available) on the NW upright of the 1X1 rack. This is for the AS WFS. Seems to work, all outlets get +/- 24 V DC.
The FSS_RMTEMP channel is very noisy after this work. I'll look into it, but probably some Acromag grounding issue.
In the afternoon, Jordan and I also laid out 4x SMA LMR240 cables and 1x DB15 M/F cable from 1X2 to the NE corner of the AP table via the overhead cable trays.
Since we will have several new 1U / 2U aLIGO style electronics chassis installed in the racks, it is desirable to have a more compact power distribution solution than the fusable terminal blocks we use currently.
I did a quick walkaround of the lab and the electronics rack today. I estimate that we will need 5 units of the 24 V and 5 units of the 18 V power strips. Each end will need 1 each of 18 V and 24 V strips. The 1Y1/1Y2/1Y3 (LSC/OMC/BHD sus) area will be served by 1 each 18 V and 24 V. The 1X1/1X2 (IOO) area will be served by 1 each 18 V and 24 V. The 1X5/1X6 (SUS Shadow sensor / Coil driver) area will be served by 1 each of 18 V and 24 V. So I think we should get 7 pcs of each to have 2 spares.
Most of the chassis which will be installed in large numbers (AA, AI, whitening) supports 24V DC input. A few units, like the WFS interface head, OMC driver, OMC QPD interface, require 18V. It is not so clear what the input voltage for the Satellite box and Coil Drivers should be. For the former, an unregulated tap-off of the supply voltage is used to power the LT1021 reference and a transistor that is used to generate the LED drive current for the OSEMs. For the latter, the OPA544 high current opamp used to drive the coil current has its supply rails powered by again, an unregulated tap-off of the supply voltage. Doesn't seem like a great idea to drive any ICs with the unregulated switching supply voltage from a noise point of view, particularly given the recent experience with the HV coil driver testing and the PSRR, but I think it's a bit late in the game to do anything about this. The datasheet specs ~50 dB of PSRR on the negative rail, but we have a couple of decoupling caps close to the IC and this IC is itself in a feedback loop with the low noise AD8671 IC so maybe this won't be much of an issue.
For the purposes of this discussion, I think both Satellite Amp and Coil Driver chassis can be driven with +/- 24 V DC.
On a side note - after the upgrade will the "Satellite Amplifiers" be in the racks, and not close to the flange as they currently are? Or are we gonna have some mini racks next to the chambers? Not sure what the config is at the sites, and if the circuits are designed to drive long cables.
DC power supplies for the RF generation box are now in place. They are the top two of the 6 Sorensens in the OMC short rack next to 1X2.
We made the connections as we did for the RF distribution box, the power supplies labele, and the cables strain-relieved.
The power supply is not yet connected to the actual RF generation box. This should be done by Suresh or someone with the supervision of him.
We have two +18V supply on the short OMC rack, in total. One is for the RF source, the other is for the OMC PZTs, whitening, etc.
This is to avoid unnecessary ground loop although the grounding situation of the OMC side is not known to me.