 Wed Aug 5 10:56:07 2015, ericq, Update, CDS, Many models crashed 
|
Last night around 1AM, many of the the frontend models crashed due to an ADC timeout. (But none of the IOPs, and all the c1lsc models were fine.)
First,
on c1sus (Wed Aug 5 00:56:46 PDT 2015) |
|
Tue Sep 1 02:44:44 2015, Koji, Summary, CDS, c1oaf, c1mcs modified for the IMC angular FF
|
[Koji, Ignacio]
In order to allow us to work on the IMC angular FF, we made the signal paths from PEM to MC SUSs.
In fact, there already were the paths from c1pem to c1oaf. So, the new paths were made from c1oaf to c1mcs. (Attachment 1~3) |
|
Thu Sep 3 02:12:08 2015, rana, Update, CDS, Simulink Webview updated
|
Back in 2011, JoeB wrote some entries on how to automatically update the Simulink webview stuff.
Somehow, the cron broke down over the years. I reran the matlab file by hand today and it worked fine, so now you can see the up to date models
using the internet. |
|
Thu Sep 3 02:30:46 2015, rana, Update, CDS, c1cal time reduced by deleting LSC sensing matrix
|
I experimented with removing somethings here and there to reduce the c1cal runtime. Eventually I deleted the LSC Sensing Matrix from it.
Ever since the upgrade, the c1cal has gone from 60 to 68 usec run time, so its constantly over.
Back when Jenne set it up
back in Oct 2013, it was running at 39 usec.
The purely CAL stuff had some wacko impossible |
 Thu Sep 3 13:25:40 2015, rana, Update, CDS, Simulink Webview updated
|
added the cron script for this to megatron to run at 8:44 AM each morning. Here's the new MegaCron attached :-()-
** it takes ~13 minutes to complete on megatron |
|
Fri Sep 4 00:58:29 2015, rana, Update, CDS, soldering the Generic Pentek interface board
|
Q and Ignacio were taking a second look at the Pentek interface board which we're using to acquire the POP QPD, ALS trans, and MCF/MCL channels.
It has a differential intput, two jumper able whitening stages inside and some low pass filtering.
I noticed that each channel has a 1.5 kHz pole associate with each 150:15 whitening stage. It also has 2 2nd order Butterworth low pass at 800 |
|
Fri Sep 4 08:00:49 2015, Ignacio, Update, CDS, RC low pass circuit (1s stage) of Pentek board
|
Here is the transfer function and cutoff frequency (pole) of the first stage low pass circuit of the Pentek whitening board.
Circuit:
|
|
Fri Sep 4 09:23:32 2015, Ignacio, Update, CDS, Modified Pentek schematic
|
Attached is the modifed Pentek whitening board schematic. It includes the yet to be installed 1nF capacitors and comments.
|
|
Fri Sep 4 20:42:14 2015, gautam, rana, Update, CDS, Checkout of the Wenzel dividers
|
Some years ago I bought some dividers from Wenzel. For each arm, we have x256 and a x64 divider. Wired in series, that means we can divide each IR beat
by 2^14.
The highest frequency we can read in our digital system is ~8100 Hz. This corresponds to an RF frequency of ~132 MHz which as much as the BBPD |
|
Tue Sep 29 03:14:04 2015, gautam, Update, CDS, Frequency divider box
|
Earlier today, the front panels for the 1U chassis I obtained to house the Wenzel dividers + RF amplifiers arrived, which meant that finally I had everything
needed to complete the assembly. Pictures of the finished arrangement attached.
Summary of the arrangement: |
|
Sun Oct 4 12:07:11 2015, jamie, Configuration, CDS, CSD network tests in progress
|
I'm about to start conducting some tests on the CDS network. Things will probably be offline for a bit. Will post when things are back
to normal. |
|
Sun Oct 4 14:23:42 2015, jamie, Configuration, CDS, CSD network test complete
|
I've finished, for now, the CDS network tests that I was conducting. Everything should be back to normal.
What I did:
I wanted to see if I could make the EPICS glitches we've been seeing go away if I unplugged everything from the CDS martian switch in 1X6 |
|
Sun Oct 4 14:32:49 2015, jamie, Configuration, CDS, CSD network test complete
|
Here's an example of the glitches we've been seeing, as seen in the StripTool trace of the front end oscillator:
You can clearly see the glitch at around T = -18. Obviously during non-glitch times the sine wave is nice and cleanish (there are still |
|
Fri Oct 9 19:54:58 2015, gautam, Update, CDS, Frequency divider box - installation in 1X2 rack
|
The new ZKL-1R5 RF amplifier that Steve ordered arrived yesterday. I installed this in the frequency divider box and did a quick check using the Fluke
RF signal generator and an oscilloscope to verify that both the X and Y paths were working.
I've now installed the box in the 1X2 rack where the olf "RF amplifiers for ALS and FOL" box used to sit (I swapped that out as |
|
Mon Oct 12 17:04:02 2015, gautam, Update, CDS, Frequency divider box - further tests
|
I carried out some more tests on the digital frequency counting system today, mainly to see if the actual performance mirrors the expected systematic
errors I had calculated here.
Setup and measurement details: |
|
Tue Oct 13 17:04:54 2015, ericq, Update, CDS, CDS things
|
After some discussion at last week's 40m meeting, I increased the frequency of daqd trying to write out minute trends from hourly to every two hours.
This has eliminated the hourly crashes. daqd still crashes sometimes, but only a few times per day. |
|
Wed Oct 14 17:40:50 2015, gautam, Update, CDS, Frequency divider box - further tests
|
Summary:
I carried out some further diagnostics and found some ways in which I could optimize the zero-crossing-counting algorithm, such that the error
in the measured frequency is now entirely within the expected range (due to a +-1 clock cycle error in the counting). We can now determine frequencies |
|
Mon Oct 19 16:16:01 2015, ericq, Update, CDS, C1ALS crashes
|
Gautam was working on his digital frequency counter stuff when the c1als model crashed. I had trouble bringing it back until I realized that, for reasons
unknown to me, the safe.snap file that the model looks for at boot had been deleted. (This
file lives in /target/c1als/c1alsepics/burt). I copied over this morning's version |
|
Tue Oct 20 17:36:01 2015, gautam, Update, CDS, Frequency counting with moving average
|
I'm working on setting up a moving-average in the custom C code block that counts the zero crossings to see if this approach is able to mitigate
the glitchy frequency readout due to mis-counting by one clock cycle between successive zero crossings. I'm storing an array the size of the moving
average window of frequency readouts at each clock cycle, and then taking the arithmetic mean over the window. By keeping a summing variable that updates |
|
Fri Oct 23 18:36:48 2015, gautam, Update, CDS, Frequency counting - workable setup prepared
|
I've made quite a few changes in the software as well as the hardware of the digital frequency counting setup.
The main change on the software side is that the custom C code that counts intervals between successive zero crossings and updates the
frequency output now has a moving average capability - the window size is readily changable (by a macro in the first line of the code, which resides at |
|
Fri Oct 23 19:27:19 2015, Koji, Update, CDS, Frequency counting - workable setup prepared
|
It is a nice scan. I'm still thinking about the equivalence of the moving average and the FIR low pass that I have mentioned in the meeting.
Anyways:
I'm confused by the plot. The bottom axis says "green beat frequency". |
|
Sat Oct 24 12:34:43 2015, gautam, Update, CDS, Frequency counting - workable setup prepared
|
Sorry for the confusion - I did mean Green beat frequency, and I had neglected the factor of 2 in my earlier calculations. However, the fit
parameter "c" in my fit was actually the half-width at half maximum and not the full width at half maximum. After correcting
for both these errors (new fit is Attachment #1, where I have now accounted for the factor of 2, and the X axis is the IR beat frequency), I don't |
|
Fri Oct 30 16:56:04 2015, gautam, Update, CDS, is there a problem with the SCHMITTTRIGGER CDS library part?
|
Over the course of my investigations into the systematic errors in the frequency readout using digital frequency counting, I noticed that my counter
variable that keeps track of the number of clock cycles between successive zero crossings was NOT oscillating between 2 values as I would have expected
(because of there being a +/- 1 clock cycle difference between successive zero crossings due to the fixed sampling time of 1/16384 seconds), but that there |
|
Wed Nov 4 16:12:07 2015, ericq, Update, CDS, OPTIMUS
|
There is a new machine on the martian network: 32 cores and 128GB of RAM. Probably this is more useful for intensive number crunching in MATLAB
or whatever as opposed to IFO control. I've set up some of the LIGO analysis tools on it as well.
A successor to Megatron, I dub it: OPTIMUS |
|
Thu Nov 5 03:04:13 2015, gautam, Update, CDS, Frequency counting - systematics and further changes
|
I've made a few more changes to the frequency counting code - these are mostly details and the algorithm is
essentially unchanged.
The C-code block in the simulink model now outputs the number of clock cycles (=1/16384 seconds) rather than the frequency itself. I've |
|
Wed Nov 11 22:50:39 2015, Koji, Configuration, CDS, Slow machine time&date
|
I was gazing at the log file for Autolocker script (/opt/rtcds/caltech/c1/scripts/MC/logs/AutoLocker.log )
and found quite old time stamps. e.g.
Old : C1:IOO-MC_VCO_GAIN 1991-08-08 14:36:28.889032 -3 |
|
Sat Nov 14 00:52:25 2015, Koji, Summary, CDS, Investion on EPICS freeze
|
[Yutaro, Koji]
Recently "EPICS Freeze" is so frequent and the normal work on the MEDM screen became almost impossible.
As a part of the investigation, all 19 realtime processes were stopped in order to see any effect on the probem. |
|
Tue Nov 17 14:31:25 2015, ericq, Update, CDS, 16Hz frame writing temporarily disabled
|
To test the effect on EPICS latency, I've restarted daqd with modified ini files which disable all frame writing of 16Hz channels.
This happened at GPS:1131835955 aka Nov 17 2015 22:52:18 UTC
Last night, I started running a script written by Dave Barker that monitors a specified EPICS channel (in this case C1:IOO-MC_TRANS_SUM), to |
|
Tue Nov 17 20:57:43 2015, ericq, Update, CDS, 16Hz frame writing running again
|
Back to nominal FB configuration at 1131857782, aka Nov 18 2015 04:56:05 UTC.
Weirdly, during this time, the script watching MC_TRANS_SUM from pianosa saw tons of freezes, but another instance watching LSC-TRY_OUT16
on optimus saw no freezes. |
|
Wed Nov 18 10:10:53 2015, ericq, Update, CDS, c1iscey IO chassis missing brackets
|
Steve and I inadvertently discovered that the c1iscey IO chassis doesn't have brackets to secure the cards where the ADC/DAC cables are
connected, making them very easy to knock loose. All other IO chassis have these brackets. Pictures of c1iscey and c1lsc IO chassis to
compare: |
|
Thu Nov 19 15:16:24 2015, ericq, Update, CDS, c1iscey IO chassis now has brackets
|
[Steve, ericq]
Brackets for the c1iscey IO chassis cards have been installed. Now, I can't unseat the cards by wiggling the ADC or DAC cable. |
|
Thu Nov 19 17:06:57 2015, Koji, Configuration, CDS, Disabled auto-launching RT processes upon FE booting
|
We want to startup the RT processes one by one on boot of FE machines.
Therefore /diskless/root/etc/rc.local on FB was modified as follows. The last sudo line was commented out.
for sys in $(/etc/rt.sh); do |
|
Wed Nov 25 14:46:53 2015, gautam, Update, CDS, c1lsc models restarted
|
I noticed that all the models running on C1LSC had crashed when I came in earlier today. I restarted all of them by ssh-ing into C1LSC and running
rtcds restart all. The models seem to be running fine now. |
|
Tue Dec 1 17:16:31 2015, gautam, Update, CDS, Script for copying BLRMS filters
|
We've been talking about putting in BLRMS filters for several channels - it would be a pain to manually copy over the correct bandpass and lowpass
filter coefficients into the newly created filter banks, and so I've set up a script (attached) that can do the job. As template filters, I'vm
using the filters rana detailed here. Essentially, what the script does is identify the |
|
Wed Dec 2 18:54:20 2015, gautam, Update, CDS, Changes to C1MCS, C1PEM
|
Summary:
I've made several changes to the C1MCS model and C1PEM model, and have installed BLRMS filters for
the MC mirror coils, which are now running. The main idea behind this test was to see how much CPU time was added as a result of setting up IPC |
|
Thu Dec 3 18:18:48 2015, gautam, Update, CDS, BLRMS for optics suspensions - library block
|
In order to be consistent with the naming conventions for the new BLRMS filters, I made a library block that takes all the input signals of interest
(i.e. for a generic optic, the coil signals, the local damping shadow sensors, and the Oplev Pitch and Yaw signals - so a total of 12 signals, unused ones
can be grounded). The block is called "sus_single_BLRMS". Inside the block, I've put in 12 BLRMS library blocks, with each |
|
Fri Dec 4 13:12:41 2015, ericq, Update, CDS, c1ioo Timing Overruns Solved
|
I had noticed for a while that the c1ioo frontend model had much higher variability than any of the other other 16k models, and would run longer than
60us multiple times an hour. This struck me as odd, since all it does is control the WFS loops. (You can see this on the Nov17
Summary page. (But somehow, the CDS tab seems broken since then, and I'm not sure why...)) |
|
Fri Dec 4 18:20:36 2015, gautam, Update, CDS, BLRMS for IMC setup
|
[ericq, gautam]
BLRMS filters have been set up for the coil outputs and shadow sensor signals. The signals are sent to the
C1PEM model from C1MCS, where I use the library block mentioned in the previous elog to put the filters in place. Some preliminary observations: |
|
Mon Dec 7 00:45:28 2015, ericq, Update, CDS, daqd is mad
|
I glanced at the summary pages and noticed that, since Friday around when we first loaded up the new BLRMS parts, daqd has crashing very frequently
(few times per hour).
I'm going to comment out the c1pem lines from the daqd master file for tonight, and see if that helps. |
|
Mon Dec 7 10:45:21 2015, ericq, Update, CDS, daqd is mad
|
Since removing c1pem from the daqd master file, daqd has not crashed. I suppose we're running into the stability issue that motivated us to disable
some of the other models (IOPs, RFM, etc.) during the RCG upgrade.
A question to Jamie: although the new framebuilder prototype still had the same problem with trend writing, can it handle this higher testpoint/DQ |
|
Mon Dec 7 11:25:10 2015, jamie, Update, CDS, daqd is mad
|
Quote:
A question to Jamie:
although the new framebuilder prototype still had the same problem with trend writing, can it handle this higher testpoint/DQ channel load? |
|
Wed Dec 9 19:01:45 2015, jamie, Update, CDS, back to fb1
|
I spent this afternoon trying to debug fb1, with very little to show for it. We're back to running from fb.
The first thing I did was to recompile EPICS from source, so that all the libraries needed by daqd were compiled for the system at hand.
I compiled epics-3.14-12-2_long from source, and installed it at /opt/rtapps/epics on local disk, not on the /opt/rtapps network mount. |
|
Mon Dec 14 23:56:29 2015, ericq, Update, CDS, c1pem reverted
|
To get C1PEM data back into the frames, I removed the new BLRMS blocks, recompiled, reinstalled, re-enabled it in daqd, restarted.
We still really want more headroom in our framebuilder situation. |
|
Tue Dec 15 11:22:53 2015, gautam, Update, CDS, c1scx and c1asx crashed
|
I noticed what I thought was excessive movement of the beam spot on ITMX and ETMX on the control room monitors, and when I checked the CDS FE status
overview MEDM screen, I saw that c1scx and c1asx had crashed. I ssh-ed into c1iscex and restarted both models, and then restarted fb as well. However,
the DAQ-DCO_C1SCX_STATUS indicator remains red even after restarting fb (see attached screenshot). I am not sure how to fix this so I am leaving it as |
|
Wed Dec 16 10:56:22 2015, gautam, Update, CDS, hard reboot of FB
|
[ericq,gautam]
Forgot to submit this yesterday...
While we were trying to get the X-arm locked to IR using MC2, frame-builder mysteriously crashed, necessitating us having to go down to the computer |
|
Thu Dec 17 14:02:05 2015, gautam, Update, CDS, IPC channels for beat frequency control set up
|
I've set up two IPC channels that take the output from the digital frequency counters and send them to the end front-ends (via the RFM model). A
summary of the steps I followed:
Set up two Dolphin channels in C1ALS to send the output of the frequency counter blocks to C1RFM (I initially used RFM blocks for these, |
|
Thu Dec 17 16:44:03 2015, gautam, Update, CDS, ALS Slow control MEDM screen updated
|
Quote:
I've not updated
the MEDM screens to reflect the two new paths yet, but will do so soon. It also remains to install appropriate filters for the servo path that takes the |
|
Wed Feb 3 08:39:17 2016, Steve, Update, CDS, blank daily summary pages
|
| Daily summary pages are blank today. Yesterday is ok |
|
Sun Mar 6 15:24:05 2016, gautam, Update, CDS, FB down again
|
I came in to check the status of the nitrogen and noticed that the striptool panels in the control room were all blank.
PMC was unlocked but I was able to relock it using the usual procedure
FB seems to be down: I was unable to ssh into it (or
any of the FEs for that matter). I checked the lights on the RAID array, they are all green. I am holding off on doing a hard reboot of FB in case there |
|
Mon Mar 7 20:40:02 2016, ericq, Update, CDS, FB down again
|
We went and looked at the monitor plugged into FB. All kinds of messages were being spammed to the screen (maybe RAM errors), and nothing could
be done to interrupt. Sadly, a hard reboot of FB was neccesary.
Video of error messages: https://youtu.be/7rea_kokhPY |