My ls plant compiled!! The RCG code can now be found in /opt/rtcds/rtscore/tags/advLigoRTS-2.5. I uploaded a copy of c1lsp.mdl onto the svn.

The weird "failed to connect" error was due to the fact that I named my inputs the same thing as my goto/from tags, so the RCG got confused. Once I renamed my inputs, it worked! I'm not sure what happened to the original "not enough parts" error; it didn't appear a single time during the rebuilding process. Anyway, I made the PDH block much neater, though the lines between PDH and ADC are looking wonky (this is purely an aesthetic problem, not a "oh god my simulation will DIE right now if I don't fix it" problem). I'll fix it in the morning; screenshot attached!

The original c1lsp was kind of sad. I updated it extensively and brought it into the modern era with color. The original c1lsp.mdl should also be on the svn. Tommorow, I'll get started on figuring out how to get LIGO specific noises from white noise.

The parts Jenne and I ordered arrived today, so we made a long cable for Guralp 1 using a 24 + 1 wire 70 meter long cable, a female 37-pin DSub, and a 26-pin milspec. The pin map is the same as the one I specified in my previous E-log. I soldered both the milspec attachment and the DSub attachment, and used a Multimeter to check the connectivity of the cables. 20 of 20 connections worked (beeped), so I plugged  the cable into the Gurlap 1 seismometer and the Guralp box.

The time series comparison for the two cables

Old cable:

New cable: (I had to move GUR 1, so it's still stabilizing in the X and Y time series)

 New

The current signal spectrum

The BLRMS on the seismic strip also look similar using the two cables - it's more visible on the wall, but I will include a StripTool picture:

New Cable BLRMS (similar to old cable BLRMS)

I revived the ETMX simplant model, c1spx.  It's running on cpu4 on c1iscex, and interfaces with C1scx via SHMEM.

The channel names for the simplant suspensions will be SUP, so the channel from this model will C1:SUP-ETMX_.

Next I'll try to get the ITMX and LSC ("LSP") simplant models running so we can run a "full" cavity simulation.

Sasha has been working on LSP, so we should be ready to do something with that soon.  In the mean time she's going to fix up the SPX MEDM screens, since some channel names have been changed since it was last run.

Jamie re-redid the ASS model a few hours ago.

I have just compiled it, and restarted c1ass.  (The model from last night is currently called c1ass3.mdl)  I had to delete and re-put inthe goto and from tags for the LSC signal coming in from the shmem.  For some reason, it kept claiming that the inputs using the from tags were not connected, even when I redid the connections.  Finally deleting and dragging in new goto and from tags made the model happy enough to compile.  Whatever.  I'm going to let Jamie do the svn-ing, since he's the one who made the changes.  Before I had figured out that it was the tags, I was concerned that the shmem was unhappy, so there was no signal connecting to the input of the goto tag, and that was somehow bad....anyhow, I recompiled the LSC model to re-create the shmem sender, but that had no effect, since that wasn't the problem.

The change from last night is that now the library parts are by DoF.  There is only one matrix in each library part, before the servo filters. Now we can DC-actuate on a single mode (ETM or ITM, pitch or yaw), and see how it affects all 4 sensors (the demodulated signals from the lockins). We need to measure the sensing matrix to go from the several sensors to the servo input.

I wanted to try out the ASS tonight, but I wanted some kinds of screens thrown together so I would know what I was doing.  Turns out screens take longer than I thought.  Am I surprised? Not really. 

They're probably at the ~85% mark now, but I should be able to try out the ASS tomorrow I think.

As we do not have legs for Trillium, I was advised to use shims to adjust the levels. However, they produce extra resonance at ~30 Hz + harmonics. Coherence is lost at these frequencies.

 Brian Lantz / Dan Clark are looking around their lab to see if they forgot to ship the feet with the T-240.  They had taken the feet off to put it in a pod. 

The script estimates digital noise produces by online filters. First version of Matlab files and complied c files are in scripts/digital_noise directory.

Algorithm for 1 filter bank (max number of filters = 10):

1.        extract sos - representation from Foton file for each filter (Matlab)
2.        download data from corresponding DQ channel using NDS (Matlab)
3.        find filters that are switched on (Matlab)
4.        filter signal using Df2 and BQF with single and double precision (C)
5.        estimate digital filter noise (Matlab)
6.        calculate power spectral density and plot the result (Matlab)

More details on (2)

Often DQ channels have reduced sampling rate. In this case the script will upsample data adding zeros.

AI filter is not applied. But in the end only the frequency range (0, DQ RATE / 2) is analyzed.

More details on (3):

• input switch ON / OFF => 4 / 0
• filters 1 - 6 ON /OFF
  • 1 => 48 / 0
  • 2 => 192 / 0
  • 3 => 768 / 0
  • 4 => 3072 / 0
  • 5 => 12288 / 0
  • 6 => 49152 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

_SW2R channel value is the sum of the following numbers:

• decimation switch ON / OFF => 512 / 0
• output switch ON / OFF => 1024 / 0
• filters 7 - 10 ON /OFF
  
  • 7 => 3 / 0
  • 8 => 12 / 0
  • 9 => 48 / 0
  • 10 => 192 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

Note: as for now Matlab script assumes that input, output and decimation filters are switched ON and there are no turned ON filter switches that do not correspond to any filters

More details on (5)

Digital noise using double precision is estimated by extrapolation of digital noise with single precision. The last is calculated by subtracting outputs of the filters with single and double precision. Then this noise is multiplied by 3 * 10^-7.

This extrapolation number was achieved by printf tests of the number 0.123456789012345678 with single and double precision on C. Using type 'float' variables 10 significant numbers show up, using type 'double' - 17.

I also did 'calibration tests' to achieve extrapolation number - signal was filters with an aggresive low-pass filter. At high frequencies filter output spectrum is flat => digital noise amplitude must be the same. The plot shows GUR1_X channel filtered with low-pass chebyshev type 1 filter.

However, extrapolation number is not the same for all cases. In the following example of analyzing BS_SUSPOS filter bank using extrapolation 3 * 10^-7 we get noise that is slightly overestimated. In some other examples we need to take a larger number. But in average, I think, this is a good approximation.

To avoid extrapolation problem we can use long double precision (~19 digits). I was able to do this with gcc compiler. However, in mex compiler using long double in filter calculations, I do not get any better precision then using double precision. I'll think more about it.

 After that RFM -> OAF communication through PCIE became bad again. Inside CommData2.c cache flushing is not allowed

// If PCIE comms show errors, may want to add this cache flushing
            #if 0
            if(ipcInfo[ii].netType == IPCIE)
                clflush_cache_range (ipcInfo[ii].pIpcData->dBlock[sendBlock][ipcIndex].data, 16);
            #endif

As a result, a significant part of MC_F and other signals is lost during RFM -> OAF transmission (270 - 330 out of 2048 per second)

Last time when I replaced 0 for 1, it suspended SUS machine because of the code bug. Alex modified a couple of files in the old version and it started to work. Do you know if this bug is fixed in the new version?

 Earthquake test our suspensions            PRM damping restored.             Oplev servo gains turned to zero.

 The power outage did not have any effect on the vacuum system.  I had to open VM1 to the RGA because of flaky CC1 cold cathode gauge fluctuations.

We are running on the vertically positioned CC1 again.

Note: rga scans with closed VM1 are back ground scans.

ACAD files of the 40m optical layout have been updated as per the vent in Aug 2011.

Files are available at the 40m svn docs-->Upgrade12-->Opt_Layout2011.

The SRM oplev beam is clipping.

I was trying to load some filters into the ASS so that I can try it out, but for some reason the filter banks aren't working - clicking the on/off buttons doesn't do anything, filters (which exist in the .txt file generated by foton) don't load.

I've emailed cds-announce to see if anyone has any ideas.

For some reason this afternoon we've been experiencing a lot of problems with the framebuilder, and with the CDS network in general.  The framebuilder has been very unresponsive, although the daqd logs seem to indicate that things are ok.  All models will loose contact with fb for very long stretches.  Attempts to kill/restart daqd don't seem to fix the problem.

These problems seem to be associated with the general CDS network issues as well.  The network seems to become very slow, and the workstations all become very slow.  The later I assume is because of the network and that so much of the work we do is on network mounted filesystems (/opt/rtcds, /ligo, etc.).

My current speculation is that daqd on fb is doing something stupid, like trying to read or write a bunch of stuff from /frames, which is also network mounted, and that clogs up the entire network.  Some serious network debugging is going to be needed to figure out what's going on, though.

I'm afraid daqd is caught in some bad state now, though.  It's not responding to anything, and every attempt to kill it seems to bring it back into the bad state.  Hopefully I can get Alex to help me figure out what's going on tomorrow.   Maybe it will clear up on it's own tonight...

It looks like daqd is indeed caught in some bad state.  It seems to die at some point after making GPS corrections to minute trender:

...
[Mon Aug  6 19:45:13 2012] Minute trender made GPS time correction; gps=1028342727; gps%60=27
tail: `fb/logs/daqd.log' has been replaced;  following end of new file
263596
MX endpoint opened
startup file interpreter thread tid=140334118615312
calling yyparse(5, 6)
[Mon Aug  6 19:50:08 2012] ->5: #set avoid_reconnect
[Mon Aug  6 19:50:08 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 19:50:08 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 19:50:08 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 19:50:08 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 19:50:08 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 19:50:08 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 19:50:08 2012] ->5: set debug=0
[Mon Aug  6 19:50:08 2012] ->5: set log=2
[Mon Aug  6 19:50:08 2012] ->5: set zero_bad_data=0
[Mon Aug  6 19:50:08 2012] ->5: set dcu_status_check=9
[Mon Aug  6 19:50:08 2012] ->5: set controller_dcu=33
[Mon Aug  6 19:50:08 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 19:50:10 2012] finished configuring data channels
[Mon Aug  6 19:50:10 2012] ->5: configure channels begin end
Unable to find GDS node 90 system c1x00 in INI files
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
[Mon Aug  6 19:50:10 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 19:50:10 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 19:50:10 2012] ->5: set detector_name="CIT"
[Mon Aug  6 19:50:10 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 19:50:10 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 19:50:10 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 19:50:10 2012] ->5: set detector_elevation=.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 19:50:10 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 19:50:10 2012] ->5: set num_dirs = 10
[Mon Aug  6 19:50:10 2012] ->5: set frames_per_dir=225
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 19:50:10 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 19:50:10 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 19:50:10 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: start main 10
[Mon Aug  6 19:50:12 2012] main started
[Mon Aug  6 19:50:12 2012] ->5: start profiler
[Mon Aug  6 19:50:12 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 19:50:12 2012] frame saver started
[Mon Aug  6 19:50:12 2012] ->5: start frame-saver
[Mon Aug  6 19:50:13 2012] ->5: sync frame-saver
[Mon Aug  6 19:50:13 2012] ->5: start trender
[Mon Aug  6 19:50:13 2012] trender started
[Mon Aug  6 19:50:13 2012] trend frame saver started
[Mon Aug  6 19:50:13 2012] ->5: start trend-frame-saver
[Mon Aug  6 19:50:14 2012] ->5: sync trend-frame-saver
[Mon Aug  6 19:50:14 2012] minute trend frame saver started
[Mon Aug  6 19:50:14 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 19:50:14 2012] Done creating ADC structures
[Mon Aug  6 19:50:15 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 19:50:15 2012] raw minute trend frame saver started
[Mon Aug  6 19:50:15 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 19:50:15 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 19:50:15 2012] ->5: #sleep 5
[Mon Aug  6 19:50:15 2012] producer started
[Mon Aug  6 19:50:15 2012] ->5: start producer
[Mon Aug  6 19:50:15 2012] ->5: start epics dcu
[Mon Aug  6 19:50:15 2012] MX receiver thread started
[Mon Aug  6 19:50:15 2012] edcu started
[Mon Aug  6 19:50:15 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 19:50:15 2012] epics server started
[Mon Aug  6 19:50:15 2012] ->5: start listener 8087
[Mon Aug  6 19:50:15 2012] ->5: start listener 8088 1
[Mon Aug  6 19:50:15 2012] ->5: sleep 60
[Mon Aug  6 19:50:15 2012] Epics server started
[Mon Aug  6 19:50:15 2012] EDCU has 2553 channels configured; first=0

[Mon Aug  6 19:50:18 2012] Minute trender made GPS time correction; gps=1028343032; gps%60=32
...

The "tail:..." line indicates that the log was moved and replaced, which indicates a daqd restart.  As far as I know this was not manually triggered.

After the restart the same thing happens again.  About once every five minutes.

When daqd is caught in this state it can not be killed.  It's in "uninterruptable sleep" ('D' state in the top output below).  This usually indicates that it's waiting for the kernel, usually due to some missing or hung IO.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
28038 controls  20   0 4430m 2.0g  19m D    0 27.1   0:15.00 daqd                                                                                         

The memory footprint also seems to be getting big.  It's clearly trying to do something stupid that it can't handle.

I tried running daqd manually, and sure enough it segfaults after about five minutes (see log below).  I've uncommented it from /etc/inittab on fb and I'm leaving it off for now until we can figure out what's going on.

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ /opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc
263596
MX endpoint opened
startup file interpreter thread tid=139790943115536
calling yyparse(5, 6)
[Mon Aug  6 20:15:27 2012] ->5: #set avoid_reconnect
[Mon Aug  6 20:15:27 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 20:15:27 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 20:15:27 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 20:15:27 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 20:15:27 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 20:15:27 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 20:15:27 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 20:15:27 2012] ->5: set debug=0
[Mon Aug  6 20:15:27 2012] ->5: set log=2
[Mon Aug  6 20:15:27 2012] ->5: set zero_bad_data=0
[Mon Aug  6 20:15:27 2012] ->5: set dcu_status_check=9
[Mon Aug  6 20:15:27 2012] ->5: set controller_dcu=33
[Mon Aug  6 20:15:27 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 20:15:30 2012] finished configuring data channels
[Mon Aug  6 20:15:30 2012] ->5: configure channels begin end
GDS server NODE=19 HOST=c1iscex DCUID=19
GDS server NODE=20 HOST=c1sus DCUID=20
GDS server NODE=21 HOST=c1sus DCUID=21
GDS server NODE=22 HOST=c1lsc DCUID=22
GDS server NODE=25 HOST=c1iscex DCUID=61
GDS server NODE=28 HOST=c1ioo DCUID=28
GDS server NODE=33 HOST=c1ioo DCUID=33
GDS server NODE=34 HOST=c1ioo DCUID=34
GDS server NODE=36 HOST=c1sus DCUID=36
GDS server NODE=38 HOST=c1sus DCUID=38
GDS server NODE=39 HOST=c1sus DCUID=39
GDS server NODE=40 HOST=c1lsc DCUID=40
GDS server NODE=42 HOST=c1lsc DCUID=42
GDS server NODE=45 HOST=c1iscex DCUID=45
GDS server NODE=46 HOST=c1iscey DCUID=46
GDS server NODE=47 HOST=c1iscey DCUID=47
GDS server NODE=48 HOST=c1lsc DCUID=48
GDS server NODE=50 HOST=c1lsc DCUID=50
GDS server NODE=51 HOST=c1ioo DCUID=51
GDS server NODE=60 HOST=c1lsc DCUID=60
GDS server NODE=61 HOST=c1iscex DCUID=61
GDS server NODE=62 HOST=c1sus DCUID=62
Unable to find GDS node 90 system c1x00 in INI files
GDS server NODE=91 HOST=c1lsc DCUID=60
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
TP: node = 19, host = c1iscex, dup = 0, prog = 0x31002013, vers = 1
Initialized TP interface node=19, host=c1iscex
TP: node = 20, host = c1sus, dup = 0, prog = 0x31002014, vers = 1
Initialized TP interface node=20, host=c1sus
TP: node = 21, host = c1sus, dup = 0, prog = 0x31002015, vers = 1
Initialized TP interface node=21, host=c1sus
TP: node = 22, host = c1lsc, dup = 0, prog = 0x31002016, vers = 1
Initialized TP interface node=22, host=c1lsc
TP: node = 25, host = c1iscex, dup = 0, prog = 0x31002019, vers = 1
Initialized TP interface node=25, host=c1iscex
TP: node = 28, host = c1ioo, dup = 0, prog = 0x3100201c, vers = 1
Initialized TP interface node=28, host=c1ioo
TP: node = 33, host = c1ioo, dup = 0, prog = 0x31002021, vers = 1
Initialized TP interface node=33, host=c1ioo
TP: node = 34, host = c1ioo, dup = 0, prog = 0x31002022, vers = 1
Initialized TP interface node=34, host=c1ioo
TP: node = 36, host = c1sus, dup = 0, prog = 0x31002024, vers = 1
Initialized TP interface node=36, host=c1sus
TP: node = 38, host = c1sus, dup = 0, prog = 0x31002026, vers = 1
Initialized TP interface node=38, host=c1sus
TP: node = 39, host = c1sus, dup = 0, prog = 0x31002027, vers = 1
Initialized TP interface node=39, host=c1sus
TP: node = 40, host = c1lsc, dup = 0, prog = 0x31002028, vers = 1
Initialized TP interface node=40, host=c1lsc
TP: node = 42, host = c1lsc, dup = 0, prog = 0x3100202a, vers = 1
Initialized TP interface node=42, host=c1lsc
TP: node = 45, host = c1iscex, dup = 0, prog = 0x3100202d, vers = 1
Initialized TP interface node=45, host=c1iscex
TP: node = 46, host = c1iscey, dup = 0, prog = 0x3100202e, vers = 1
Initialized TP interface node=46, host=c1iscey
TP: node = 47, host = c1iscey, dup = 0, prog = 0x3100202f, vers = 1
Initialized TP interface node=47, host=c1iscey
TP: node = 48, host = c1lsc, dup = 0, prog = 0x31002030, vers = 1
Initialized TP interface node=48, host=c1lsc
TP: node = 50, host = c1lsc, dup = 0, prog = 0x31002032, vers = 1
Initialized TP interface node=50, host=c1lsc
TP: node = 51, host = c1ioo, dup = 0, prog = 0x31002033, vers = 1
Initialized TP interface node=51, host=c1ioo
TP: node = 60, host = c1lsc, dup = 0, prog = 0x3100203c, vers = 1
Initialized TP interface node=60, host=c1lsc
TP: node = 61, host = c1iscex, dup = 0, prog = 0x3100203d, vers = 1
Initialized TP interface node=61, host=c1iscex
TP: node = 62, host = c1sus, dup = 0, prog = 0x3100203e, vers = 1
Initialized TP interface node=62, host=c1sus
TP: node = 91, host = c1lsc, dup = 0, prog = 0x3100205b, vers = 1
Initialized TP interface node=91, host=c1lsc
[Mon Aug  6 20:15:30 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 20:15:30 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 20:15:30 2012] ->5: set detector_name="CIT"
[Mon Aug  6 20:15:30 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 20:15:30 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 20:15:30 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 20:15:30 2012] ->5: set detector_elevation=.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 20:15:30 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 20:15:30 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 20:15:30 2012] ->5: set num_dirs = 10
[Mon Aug  6 20:15:30 2012] ->5: set frames_per_dir=225
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 20:15:30 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 20:15:30 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 20:15:30 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 20:15:30 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 20:15:30 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 20:15:30 2012] ->5: start main 10
Allocated move buffer size 11616356 bytes
[Mon Aug  6 20:15:32 2012] main started
[Mon Aug  6 20:15:32 2012] ->5: start profiler
[Mon Aug  6 20:15:32 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 20:15:32 2012] frame saver started
[Mon Aug  6 20:15:32 2012] ->5: start frame-saver
[Mon Aug  6 20:15:33 2012] ->5: sync frame-saver
[Mon Aug  6 20:15:33 2012] ->5: start trender
[Mon Aug  6 20:15:33 2012] trender started
[Mon Aug  6 20:15:33 2012] trend frame saver started
[Mon Aug  6 20:15:33 2012] ->5: start trend-frame-saver
[Mon Aug  6 20:15:34 2012] ->5: sync trend-frame-saver
[Mon Aug  6 20:15:34 2012] minute trend frame saver started
[Mon Aug  6 20:15:34 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 20:15:34 2012] Done creating ADC structures
[Mon Aug  6 20:15:35 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 20:15:35 2012] raw minute trend frame saver started
[Mon Aug  6 20:15:35 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 20:15:35 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 20:15:35 2012] ->5: #sleep 5
[Mon Aug  6 20:15:35 2012] producer started
[Mon Aug  6 20:15:35 2012] ->5: start producer
[Mon Aug  6 20:15:35 2012] ->5: start epics dcu
[Mon Aug  6 20:15:35 2012] MX receiver thread started
[Mon Aug  6 20:15:35 2012] edcu started
[Mon Aug  6 20:15:35 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 20:15:35 2012] epics server started
[Mon Aug  6 20:15:35 2012] ->5: start listener 8087
[Mon Aug  6 20:15:35 2012] ->5: start listener 8088 1
[Mon Aug  6 20:15:35 2012] ->5: sleep 60
Creating C1:DAQ-DC0_PEM_SLOW_STATUS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_CPS
Creating C1:DAQ-DC0_PEM_SLOW_CRC_SUM
Creating C1:DAQ-DC0_C1X01_STATUS
Creating C1:DAQ-DC0_C1X01_CRC_CPS
Creating C1:DAQ-DC0_C1X01_CRC_SUM
Creating C1:DAQ-DC0_C1X02_STATUS
Creating C1:DAQ-DC0_C1X02_CRC_CPS
Creating C1:DAQ-DC0_C1X02_CRC_SUM
Creating C1:DAQ-DC0_C1SUS_STATUS
Creating C1:DAQ-DC0_C1SUS_CRC_CPS
Creating C1:DAQ-DC0_C1SUS_CRC_SUM
Creating C1:DAQ-DC0_C1OAF_STATUS
Creating C1:DAQ-DC0_C1OAF_CRC_CPS
Creating C1:DAQ-DC0_C1OAF_CRC_SUM
Creating C1:DAQ-DC0_C1ALS_STATUS
Creating C1:DAQ-DC0_C1ALS_CRC_CPS
Creating C1:DAQ-DC0_C1ALS_CRC_SUM
Creating C1:DAQ-DC0_C1X03_STATUS
Creating C1:DAQ-DC0_C1X03_CRC_CPS
Creating C1:DAQ-DC0_C1X03_CRC_SUM
Creating C1:DAQ-DC0_C1IOO_STATUS
Creating C1:DAQ-DC0_C1IOO_CRC_CPS
Creating C1:DAQ-DC0_C1IOO_CRC_SUM
Creating C1:DAQ-DC0_C1MCS_STATUS
Creating C1:DAQ-DC0_C1MCS_CRC_CPS
Creating C1:DAQ-DC0_C1MCS_CRC_SUM
Creating C1:DAQ-DC0_C1RFM_STATUS
Creating C1:DAQ-DC0_C1RFM_CRC_CPS
Creating C1:DAQ-DC0_C1RFM_CRC_SUM
Creating C1:DAQ-DC0_C1PEM_STATUS
Creating C1:DAQ-DC0_C1PEM_CRC_CPS
Creating C1:DAQ-DC0_C1PEM_CRC_SUM
Creating C1:DAQ-DC0_C1X04_STATUS
Creating C1:DAQ-DC0_C1X04_CRC_CPS
Creating C1:DAQ-DC0_C1X04_CRC_SUM
Creating C1:DAQ-DC0_C1LSC_STATUS
Creating C1:DAQ-DC0_C1LSC_CRC_CPS
Creating C1:DAQ-DC0_C1LSC_CRC_SUM
Creating C1:DAQ-DC0_C1SCX_STATUS
Creating C1:DAQ-DC0_C1SCX_CRC_CPS
Creating C1:DAQ-DC0_C1SCX_CRC_SUM
Creating C1:DAQ-DC0_C1X05_STATUS
Creating C1:DAQ-DC0_C1X05_CRC_CPS
Creating C1:DAQ-DC0_C1X05_CRC_SUM
Creating C1:DAQ-DC0_C1SCY_STATUS
Creating C1:DAQ-DC0_C1SCY_CRC_CPS
Creating C1:DAQ-DC0_C1SCY_CRC_SUM
Creating C1:DAQ-DC0_C1ASS_STATUS
Creating C1:DAQ-DC0_C1ASS_CRC_CPS
Creating C1:DAQ-DC0_C1ASS_CRC_SUM
Creating C1:DAQ-DC0_C1CAL_STATUS
Creating C1:DAQ-DC0_C1CAL_CRC_CPS
Creating C1:DAQ-DC0_C1CAL_CRC_SUM
Creating C1:DAQ-DC0_C1MCC_STATUS
Creating C1:DAQ-DC0_C1MCC_CRC_CPS
Creating C1:DAQ-DC0_C1MCC_CRC_SUM
Creating C1:DAQ-DC0_C1MCP_STATUS
Creating C1:DAQ-DC0_C1MCP_CRC_CPS
Creating C1:DAQ-DC0_C1MCP_CRC_SUM
Creating C1:DAQ-DC0_C1LSP_STATUS
Creating C1:DAQ-DC0_C1LSP_CRC_CPS
Creating C1:DAQ-DC0_C1LSP_CRC_SUM
Creating C1:DAQ-DC0_C1SPX_STATUS
Creating C1:DAQ-DC0_C1SPX_CRC_CPS
Creating C1:DAQ-DC0_C1SPX_CRC_SUM
Creating C1:DAQ-DC0_C1SUP_STATUS
Creating C1:DAQ-DC0_C1SUP_CRC_CPS
Creating C1:DAQ-DC0_C1SUP_CRC_SUM
[Mon Aug  6 20:15:35 2012] Epics server started
[Mon Aug  6 20:15:35 2012] EDCU has 2553 channels configured; first=0

Symmetricom status: LOCKED
Starting at gps 1028344552 prev_gps 1028344552 frac 312500000 f 314094022
[Mon Aug  6 20:15:38 2012] Minute trender made GPS time correction; gps=1028344552; gps%60=52
Segmentation fault (core dumped)

I'm trying to get more of the simplant models running so that we (me and Sasha Surf) can get a full real-time cavity simplant running.  As I reported last week, c1spx is running again on c1iscex.

The two new simplant models are c1sup, which holds the simplant for ITMX, and c1lsp, which holds the IFO simplant, specifically the one we're working on for XARM.

Here's the relevant info:

model  host     dcuid  cpu
c1spx  c1iscex  61     4
c1sup  c1sus    62     6
c1lsp  c1lsc    60     6

c1spx and c1sup will be running the sus_single_plant parts for ETMX and ITMX simplant.  All the simplant suspension channels will be names "SUP" (as opposed to "SUS" for control).

c1lsp is now running, but c1sup won't run for unknown reasons.  The c1sup model is not very complicated, and in fact is more-or-less identical to c1spx.  It compiles and installs and even loads, but it completely unresponsive after loading.  Unfortunately I've had enough CDS bullshit for today, so I'll try to figure out what's going on tomorrow.

The PRM was pointing totally the wrong way, so there was no light on the oplev PD.  I restored the PRM, turned the gains back to (0.15, -0.3) as per Yuta's elog 6952, and it seems just fine to me.

I want to check the data from last night / the weekend to see when the mispointing happened, but dataviewer can't connect to the fb, since Jamie is still working his magic.  I'm pretty sure I restored all of the optics after Eric finished playing with MICH Friday night, but it's possible that I forgot one, I suppose.  If it wasn't me, then I'm curious when it happened.

When the network / fb went bad this afternoon, I had been working on the ASS model, shortening the names of the filter banks to fix the problem from elog 7092.  I wanted to finish working on that, so the ASS model is now rebuilt with slightly shorter names in the filterbanks (which fixes the problem of the filter banks not working).

------------------------------------------------------------

I mentioned this to Jamie the other day, but here's the error that you get when the GoTo/From tags aren't working:

>>rtcds make c1ass
### building c1ass...
Cleaning c1ass...
Done
Parsing the model c1ass...
IPC 9 8 is C1:LSC-ASS_LSC
IPC 9 8 is ISHME
IPC 10 9 is C1:RFM-LSC_TRX
IPC 10 9 is IPCIE
IPC 11 10 is C1:RFM-LSC_TRY
IPC 11 10 is IPCIE

INPUT XARM_LSC_in is NOT connected

INPUT YARM_LSC_in is NOT connected

***ERROR: Found total of ** 2 ** INPUT parts not connected

make[1]: *** [c1ass] Error 255
make: *** [c1ass] Error 1

I don't know why these tags weren't working, but there was a GoTo tag on the output of the LSC shmem block, and then Froms on each of the XARM_LSC_in and YARM_LSC_in.  The other day I played around with a bunch of different things (grounding inputs, terminating outputs, whatever), but finally replacing the tags with identical ones freshly taken from CDS_PARTS made it happy. 

I wrote new setup, on and off scripts for the arm ass.  They take the arm as an argument, so it's the same script for both arms.  Scripts are in ...../scripts/ASS/ , and have been checked in to the 40m svn.

So far the on script doesn't really do anything, since I haven't chosen values for the CLKGAINs of the lockins.  The old values were 30 for lockins 12, 14, 27, 29 and 250 for lockins 7, 9, 22, 24.  Unfortunately, I have no memory of which lockin means what in the old numbered system.  I'll have to look that up somehow.  Or, just dither the optics using some value and look at the spectrum to see the resulting SNR and just pick something that gives me reasonable SNR.

I modified the ASS model slightly:

* Added an overall gain to the ASS_DOF2 library part, between the matrix and the servo inputs so we can do soft startups.  Self - remember that the main ASS screen needs to be modified to reflect this!

* Rearranged the order that the demodulated signals go into the matrix.  I hadn't paid attention, and the old ordering had the transmission (TRX/TRY) demod signals interleaved with the LSC demod signals.  I've changed it to be all the TR signals, then all the LSC signals.  I think this makes more sense, since we will use these inputs separately, so now they're on different halves of the matrix.

Alex is apparently working on daqd (remotely).  I'll report back when I find out more.

So daqd's problem was apparently the bad/non-running c1sup model.  The c1sup model, which I reported on attempting to get running in 7097, was not running because there were no available CPUs on the c1sus FE machine.  This was due to my stupid undercounting of the number of CPUs.  Anyway, for reasons I don't understand, this was causing daqd to segfault.  Removing c1sup from c1sus "fixed" the problem.

Alex agreed that daqd should definitely not be segfaulting in this circumstance.  It's still unclear exactly what daqd was looking at that was causing it to crash.

I'm going to move c1sup to c1iscex, which has a lot of spare CPUs.

I spoke too soon.  It's still segfaulting, but at a different place. Alex and I are looking into it.

But another mystery solved is the cause of all the network slowness: the daqd core dump.  When daqd segfaults it dumps it's core, which can typically be >4G, to /opt/rtcds/caltech/c1/target/fb/core.  This is of course an NFS mount from linux1, so it's dumping 4G on the network, which not surprisingly clogs the network.

Previously, medmrun didn't accept arguments to pass along to the script it was going to run.  Jamie has graciously taken a moment from fixing the computer disaster to help me update the medmrun script.

Now the ASS scripts are call-able from the screen.

Apparently the last problem was because of root-owned frame directories that daqd was trying to write to.  During debugging Alex had run daqd as root, but it's supposed to run as controls.  All the /frame directories are supposed to be owned by controls.  When daqd was run as root, it created new frame directories owned by root, which controls couldn't write to when I restarted daqd the proper way.  Once we chown'd the directories daqd started running again.

Alex also put in a "fix" for the core dump problem.  He touched an empty core file owned by root:

-rw-r--r-- 1 root root 0 Aug  7 14:38 /opt/rtcds/caltech/c1/target/fb/core

This will prevent any dying daqd process owned by controls from dumping it's core at that location.  Personally I think this is a horribly hacky "solution" that doesn't actually fix any of the issues that were causing the segfaults to begin with, but it might prevent some of the network slow down we see when the core does dump.  It's mostly just masking the problem, though, so I'm tempted to remove it so we all feel the pain when daqd starts shitting all over the network again.

 Sanwiched wall as shown: 1" clear acrylic, 2 layers of 0.004" thick "window tint", 1 layer of  0.007" thermashield and  0.125" yellow acrylic

Visibility: 70 %,    Transmission of 1064 nm  2-3 % at 0-50 degrees incident  power density 0.7 W/mm2

Max power 100 mW

 Good. The power density and max power are not important (especially since you don't define a quantitative way to spec them). We ONLY care about the transmission.

Please check the summary pages out at the link below and let me know if there are any modifications I should make!  All existing pages are up to date and contain all of the pages I have.

Questions, comments, and suggestions will be appreciated! Contact me at endavison@umail.ucsb.edu

https://nodus.ligo.caltech.edu:30889/40m-summary/

Yesterday I plugged the geophone and accelerometer output into the ADC, rather than the SR785, so I could collect for longer and take more data at once.

As per Rana's suggestion, I am also now taking the geophone output after the first op-amp in the circuitry following the geophone (a low-noise op-amp, OPA227). It acts as a buffer so I'm not just measuring other local noise sources (which explains why the geophone noise curve sort of matched the SR785 noise curve in my old plots).

With these changes, I remeasured the accelerometer and geophone noises as well as collected an ASD of a geophone sitting on the STACIS in open loop operation. I also looked up the noise specs for the various op-amps in the geophone pre-amp and high voltage board; everything I found, I added in quadrature to come up with an approximate op-amp noise value for the STACIS. All of this is plotted below:

I left the y-axis in V/rtHz instead of converting it to m/s/rtHz so that the op-amp noise could be compared to the other noises. All sensor data was taken with the sensors horizontal (noise data taken in granite and foam).

The accelerometer and geophone noise still appear to be similar, and the op-amp noise, at least according to specs, is low compared to the other noises. This implies there's not much to gain from switching the geophones with accelerometers nor with swapping out the op-amps for lower-noise components (unless the ones I couldn't find specs for were high-noise, though it seems like mainly low-noise components were used). 

 Just felt an EQ. Impulse moved some vertical blinds by several mm.

Tue Aug 07 23:26:06 2012 

 All optics except MC2 and ETMX are crazy

Looks like you're just measuring the ADC noise. You should add ADC noise to your plot. To compare the geophones with the accelerometers, you have to correct for the preamp gain and plot them both in the same units.

To get above the ADC noise you can use an SR560 preamp. (AC Coupled, G = 100)

[Sasha, Masha, Liz, Eric]

A bunch of surfs in the lab just noticed that ETMX is going crazy (laser is shifting everywhere) due to a 4.5 EQ that just hit LA. The optic is already shut down according to the watchdogs.

There were another couple of earthquakes at about 9:30am and 9:50am local.

All but MC2 were off the watchdogs.  I damped and realigned everything and everything looks ok now.

Over the past week, I have been working on my progress report and finalizing the summary pages.  I have a few more things to address in the pages (such as starting at 6 AM, including spectrograms where necessary and generating plots for the days more than ~a week ago) but they are mostly finalized.  I added all of the existing acoustic and seismic channels so the PEM page is up to date.  The microphone plots include information about the transfer factor that I found on their information sheet (http://www.primomic.com/).  If there are any plots that are missing or need editing, please let me know!

I also modified the c1_summary_page.sh script to run either the daily plots or current updating plots by taking in an argument in the command line.  It can be run ./c1_summary_page.sh 2012/07/27
 or ./c1_summary_page.sh now to generate the current day's pages.  (Essentially, I combined the two scripts I had been running separately.)  I have been commenting my code so it is more easily understandable and have been working on writing a file that explains how to run the code and the main alterations I made.  The most exciting thing that has taken place this week is that  the script went from taking ~6 hours to run to taking less than 5 minutes.  This was done by using minute trends for all of the channels and limiting the spectrum plot data.

The summary pages for each day now contain only the most essential plots that give a good overview of the state of the interferometer and its environment instead of every plot that is created for that day.

I am waiting for Duncan to send me some spectrogram updates he has made that downsample the timeseries data before plotting the spectrogram.  This will make it run much more quickly and introduce a more viable spectrogram option.

Today's Summary Pages can be accessed by the link on the wiki page or at:

https://nodus.ligo.caltech.edu:30889/40m-summary/archive_daily/20120808/

Hey, the pages got significantly nicer than before. I will continue to give you comments if I find anything.

So far: There are many 10^-100 in logarithmic plots. Once they are removed, we should be able to see the seismic excitation during these recent earth quakes?

Incidentally, where the script is located? "./" isn't the absolute path description.

Jan and Manasa are going to elog about their work later, but it involved putting a BS/window/some kind of pick off in front of the MC Trans QPD, so the total light on the MC Trans QPD is now ~16000 rather than ~26000 counts.  I changed the threshold in the MC autolocker to 5000, so now the MC Trans PD must see at least 5000 counts before the autolocker will engage the boosts, WFS, etc.  Actually, this threshold I believe should have been some several thousand value, but when I went in there, it was set to 500 counts, for low power MC mode during a vent.  It had never gotten put back after the vent to some higher, nominal value.

I turned on the ASS, without closing the loops, to try to measure the sensing matrix. 

The Yarm was locked (Eric did a nice job earlier - he'll ELOG ABOUT IT before he goes home!), and I used an LO CLKGAIN of 300 on all of the TRY Lockins.  Then I put on and took away a 10% offset in pitch, but it's almost impossible to see the difference. 

The attached is a truly awful screenshot, but you can kind of see what's going on.  The big steps are me increasing the LO gain, but around "0" on the x-axis I changed the pitch offset from 10% away to nominal.  Since there are such big oscillations, the change is basically non-existent.  Grrrr. I'll look at it again tomorrow, since I have an exiting bike ride home ahead of me....

From the log, I couldn't understand what has been done.

The procedure we should perform is

1. Dither total 4 dofs of the two mirrors with different frequencies. Some fluctuation of TRY is even visible in dataviewer.
2. The cavity is aligned at the beginning. These 4 peaks in TRY in DTT is small or invisible. Some 2nd harmonics are visible.
3. Misalign one of the dofs. Some peaks get bigger.
4. Correspoding lockin output becomes bigger.

Then you can start measuring the sensing matrix. At which part did the attempt fail?

 To ease the pain of hunting files, optical layout ACAD files have been moved to a new directory 40M_Optical Layout in the repository. Relevant files from directories Upgrade12 and upgrade 08 will be moved to "40M_Optical Layout" very soon and eventually these old directories will be removed. 

 Cavity started out aligned pretty well, but not 100%.  Transmission was ~0.8 . Perhaps this was part of the problem.

I realize now that you mention it, it was totally amateur hour of me to only look at the lockin outputs on StripTool (plus POY and TRY on Dataviewer), and not look at TRY on DTT...or any spectra at all.  Not so intelligent.  I could see some fluctuation of TRY on Dataviewer that corresponded to me turning on the oscillators, as well as the spot wiggling on the camera view of ETMYT a teeny bit.

When applying a 10% misalignment to ETMY Pit (by adding 0.1 to the Pit components of the output matrix, as is done in the MC spot position calibration), I could see that there was a small jump in the StripTool trace, but it was much smaller than the ambient fluctuations of the output. 

I just looked back and realized that I must have forgotten to add my screenshot, but it's saved on a desktop on Rossa.  It would be better if I had attached the data, but from that you see that the average of the lockin output signal didn't change very much in the last several minutes of the measurement, but the fluctuations (no misalignment offsets) are pretty big, maybe ~10% or 15% the size of the signal.  Then when I added the misalignment to one mirror (ETMY PIT), there is a very small jump in the lockin signal, but it is much, much smaller than the size of the ambient fluctuations.  Perhaps a long average would result in a "real" value, but by looking by eye, I can't see a discernible difference in the average value of the lockin outputs.

My plan is to do as you say, dithering all 4 optics, and misaligning a single optic's single DoF (Pit or Yaw), and seeing how that misalignment affected each of the sensors (the lockin outputs).  Then put that DoF back to nominal, and misalign a different DoF, rinse and repeat.

Okay, so this is a little stream-of-consciousness-y, and you're going to think I'm really dumb, but I just realized that I haven't set the phase of the lockin demodulators yet.  So I think I need to dither the optics, and go through each of the sensors, and adjust the phase until the peak in TRY in DTT is maximized for the I phase, and minimized for the Q phase (since we use the I-output).  Bah.  Bad Jenne.

That's a good point, but I suspect that you end up with the in-phase (0deg) as the response of the IFO is immediate compared with the dithering frequency
as long as the whitening/dewhitening are properly compensated in the digital realm.

As the subject states, all screens are working (including the noise screens), so we can keep track of everything in our model! :D I figured out that I was just getting nonsense (i.e. white noise) out of the sim plant cause the filter matrix (TM_RESP) that controlled the response of the optics to a force (i.e. outputted the position of the optic DOF given a force on that DOF and a force on the suspension point) was empty, so it was just passing on whatever values it got based on the coefficients of the matrix without DOING anything to them. In effect, all we had was a feedback loop without any mechanics.

I've been working on getting the mechanics of the suspensions into a filter/transfer function form; I added something resembling that into foton and turned the resulting filter on using the shiny new MEDM screens. However, the transfer functions are a tad wonky (particularly the one for pitch), so I shall continue working on them. It had a dramatic effect on the power spectrum (i.e. it looks a lot more like it should), but it still looks weird.

Still haven't found the e-log Jamie and Rana referred me to, concerning the injection of seismic noise into the simulation. I'm not terribly worried though, and will continue looking in the morning. Worst case scenario, I'll use the filters Masha made at the beginning of the summer.

Masha and I ate some of Jamie's popcorn. It was good.

 Okay! Attached are two power spectra. The first is a power spectrum of reality, the second is a power spectrum of the simPlant. Its looking much better (as in, no longer obviously white noise!), but there seems to be a gain problem somewhere (and it doesn't have seismic noise). I'll see if I can fix the first problem then move on to trying to find the seismic noise filters.

The elog was not responding. I attempted to restart it by running .../start-elog.csh, but this didn't work (even after the usual ~2 times it needs).

Somehow pkill did not kill the daemon, so I kill -9'd it and that did the trick. I ran the start script once more and it came back online.

Koji pointed out that I was being silly, and rather than actually misaligning the optics (by, say, changing their IFO Align sliders) I was changing the location of the actuation node by changing the coil output gains.  Now I see nice signals at the I_OUT of each of the demodulators (so far I've only looked at the YARM).

I've measured and inverted the matrix by taking the nominal values of the demodulator outputs when the optics are all by-hand optimally aligned, then one-by-one misaligning an optic's angle (pitch or yaw), and looking at the demod outputs that result.  Repeat with each misalignment DoF for each of the 4 rows of the matrix.  Then I set the pit/yaw coupling elements of the matrix to zero.  Then invert the matrix, put it in, and see what happens.  So far, the yaw DoFs converged to zero, but the pitch ones didn't.  I'll play with it more and think some more tomorrow.