Still not able to resolve the issue.

Except for c1lsc, the models are not running on any of the FE machines . I can ssh into all the machines but could not restart the models on the FE using the usual rtcds restart <modelname>

Something happened around 4AM (inferring from the Striptool on the wall) and the models have not been running since then.

[Rana, Mirko]

I tried out the virtual pendulum idea today. The idea is to compensate the physical pendulum via the control system, and then add a virtual pendulum formed in the control system. We know the actuator TF from p. 5900 and apply its inverse to the C1:SUS-MC?_SUSPOS filters. Also we add an pendulum Q=3 with a resonance frequency of 0.1Hz to the POS control loops.

The result is pretty awesome!

1. Black: Standard config. Wfs on. New Cheby filter in place ( p. 6031 )
2. Red: With virtual pendulum. Extra eliptic LP filter @ 2.5Hz

Filter shape:

This is stable for 5-10minutes, at which point it falls out of lock, swinging in yaw.

In the above entry MC_f  signal is improved off of resonance by the implementation of the pendulum compensation. It should be checked if this is actually due to the implementation of the virtual pendulum at 0.1Hz. A way to check that might be to implement a simple double LP at 0.1Hz instead and look at the resulting MC_f signal. A projection of the OSEM FB noises with the compensation active might be interesting.
The physical resonance at 1Hz is clearly not compensated correctly, which is probably the reason for the lock losses after a few minutes. It might be a good start to measure the residual resonance with the compensation in place. The filters in bank 7 of C1:SUS-MC?_SUSPOS have both the compensation of the 1Hz resonance( really the inverse actuator TF ) and the virtual pendulum in them. The ‘pure’ compensation can be found in the InvTF module in the C1:OAF-ADAPT_MCL_CORR filter.
The fact that the beam swings in yaw at lock loss indicates that the separation of the DOFs might not be perfect. One should have a look into the yaw and pitch DOFs with the compensation active.

This totally creeped me out when I found it wandering around on the floor not so far from my desk:

The End IO chassis have small trenton boards, which apparently only have 5 usuable PCI slots, even though there are 6 on the board.  This is because of the way the the host interface board is setup and its closeness to the 2nd to last PCI slot

The PMC to PCIe adapters I was handed by Jay for use with the RFM cards require a 4 pin power connection at the top, which are not available inside the thin 1U computers.

The only solution I can come up with is swap the PMC to PCIe adapters for the RFM cards with adapters for some of the already installed ADCs and DACs which do not require power directly from the power supply.  This should make it possible to mount the RFM card in the computer, at least for the ends.  Since the SUS and IOO chassis will have more slots available than needed, the RFM cards can be slotted into those.  The SUS has to fit in the chassis since the computer will have the Infiband host adapter and a dolphin connector for talking to the LSC machine.

There is still the problem of actually getting the RFM card into the computer, but that should be possible with a little bit of bending of the left side of the computer frame.

fb:controls>./tpman &
[1] 1037
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0

A paltry two locks tonight, but not entirely useless. I had some issues keeping the PRMI locked, which some additional boosts helped with. But, my feeling was that our crossover process is not tuned well. 

At full lock, both sub-loops have high gain around the crossover region, so the usual DTT loop transfer function measurement produces a meausrement of Gdigital/G_aopath (or minus that. I.e. I'm not currently 100% which is the bad phase in this plot, though it intuitive looks like 0 ). Thus, we can directly look at the crossover frequency and the effect of the different filters there. (I've also been working on an up-to-date CARM loop model today, so this will help inform that). 

Below, the black traces are the crossover at the end of the script when using the 120:500 "helper," and purple is without it. As we turn up the AO path gain, the trace "falls" from above, which explains why we can see instabilities around the violin filter. 

Having the helper on definitely made the probability of surviving the first overall CARM gain ramp higher, but it's not currently intuitively clear to me why that is the case. Afterwards, we can turn the helper off, to keep the shallower crossover shape. This is what I've put in to the up script for now. I also added a few seconds delay for when the script wants to switch DARM to RF only; I found it was maybe speeding too fast through this point.

DTT xml attached

Here's the comparison of last night's crossover measurement to my loop model. Not stellar, but not totally off base. All of the digital filters are read directly from the foton filter file, and translated from their SOS coefficients, so they should be accurate. I may have tallied together the wrong arrangement of FMs, though. I will recheck. 

Although I don't have a measurement to compare it with yet (as I don't know where the crossover was, the filter statesolder, etc. for the older loop measurements), here's what my current CARM loop model looks like, just for kicks. Here, only the first CM board boost is on. If we turn on some super boosting, we can probably ease up on some of the digital boosts, lower the crossover frequency, and put some lowpass that suppress the violin filters' effect on the crossover and reduces digital sensing noise injection. 

Lastly, I'll just note that my current MIST model predicts that the CARM cavity pole should be at ~170Hz, and a peak arm transmission of 180 times single arm power. I saw powers of ~120 last night. 

Whoops, I implemented the IOP downsampling filters wrong. Once I did that, it looked like just delay mismatch, so I added two more computation cycles for a total of four 16k cycles, which is maybe not so justified... Nevertheless, model and measurement now agree much better. Here are the corrected plots. 

To further reduce the RF power at 110MHz in the REFL165 chain, I made a twin-t notch in a pomona box.

It is tuned at 110.66MHz.

The inductor is Coil Craft 5mm tunable (164-09A06SL 100-134nH).
Without the 10Ohm resister (like a usual notch), the dip was ~20dB. With this configuration, the notch of -42dB was realized.

Q >> Please measure the RF spectrum again with the notch.

The notch filter has been installed directly attached to the output of the SHP-150 at the PD output. Structurally, there is a right angle SMA elbow between the two filters; I set up a post holder under the notch pomona box to prevent torque on the PD. Via directional coupler and AG4395, we measured the output of the REFL165 RF amplifier with the PRMI locked on REFL33. 

Note, the plot below is not referred to the amplifier output, as in my previous plots; it is directly representative of the amplifier output spectrum. 

There are no RF signals being output above -28dBm, thus I am confident that we are not subject to compression distortion. 

Given the last measurements we made (ELOG 10692), I estimate that the notch has reduced the power at 110MHz by ~33dB, which is 9dB higher than the notch performance Koji measured when he made it. Maybe this could be due to the non-50Ohm impedance of the HPF distorting the tuning, or I physically detuned it when mounting it on the PD. Still, 33dB is pretty good, and may even give us room to amplify further. (ZRL-700+ instead of the ZFL-1000LN+?)

The new fiber beam splitters we are ordering, PFC-64-2-50-L-P-7-2-FB-0.3W, have the slow axis working and fast axis blocked. The way the light is coupled into the fibers right now is done to maximize the amount of light into the fast axis. So we will have to do a 90deg rotation if we use that part. Probably the easiest thing to do is to put a HWP immediately before the free-space-to-fiber collimator.

Update 6pm: They have an "SB" version of the part with the slow axis blocked and fast axis enabled, same price, so I'll ask Chub to get it.

Andrew seems to have an integrated solution of PBS+HWP in a singe mount. Or, I wonder if we should use HWP/QWP before the coupler. I am interested in a general solution for this problem in my OMC setup too.

Your PZT is broken.

R

We have (now) in the lab 2 cables that are RJ45-DB9.  The gray one is LIGO-made, while the blue one is store-bought.  

The gray LIGO-made one works, but the blue store-bought one does not.  I checked their pinouts, and they are completely different.  On the sketch below, the pictures of the connectors is me looking at them face-on, with the cables going out the back of the page.  The DB9 is female. 

 There are RJ45-DB9 adapters in the big spinny rack next to the linux1 rack that are for this exact purpose.  Just use a stanard ethernet cable with them.

Rana points out that we haven't had fast channels for PMC (trans, refl, pzt), input laser things, more FSS things since the upgrade.  Bad.

For my note-taking:

1. Lock PRMI with ITMs as the MICH actuator. Confirm that the MICH-->PRCL contribution cannot be nulled. ✅  [15960]
2. Lock PRMI on REFL165 I/Q. Check if transition can be made smoothly to (and from?) REFL55 I/Q.
3. Lock PRMI. Turn sensing lines on. Change alignment of PRM / BS and see if we can change the orthogonality of the sensing.
4. Lock PRMI. Put a razor blade in front of an out-of-loop photodiode, e.g. REFL11 or REFL33. Try a few different masks (vertical half / horizontal half and L/R permutations) and see if the orthogonality (or lack thereof) is mask-dependent.
5. Double check the resistance/inductance of the PRM OSEMs by measuring at 1X4 instead of flange. ✅  [15966]
6. Check MC spot centering.

If I missed any of the tests we discussed, please add them here.

I've attached my handwritten notes covering all the serial communications in the vac system, and the relevant wiring for all the adapters, etc. I'll work with Chub to produce a final documentation, but in the meantime this may be a useful reference.

Juan and I built an analog setup to measure some transfer functions of the MOS suspension. The setup is blocking the lab way around the PD test bench.
Excuse us for the inconvenience. It will be removed/cleared by the end of the week.

The setup was (at least partially) cleared.

Given some of the things we've facing lately, it occurs to me that we could be better served by having some sort of unified human-alerting scheme in place, for things like:

• Local/offsite backup failures
• Vaccumm system problems
• HDD status for things like /frames/ and /cvs/cds/, whether the disks are full, or their SMART status indicates imminent mechanical failure

Currently, many of these things are just checked sporadically when it occurs to someone to do so, or when debugging random issues. Smoother IFO operation and peace of mind could be gained if we're confident that the relevant people are notified in a timely manner. 

Thoughts? Suggestions on other things to monitor, like maybe frontend/model crashes?

> And.....we have also lost the DAQ channels that used to be associated with the _IN1 of the SUSPOS/PIT/YAW filter modules. Please put them back; our templates don't work without them.

I have (re?)added the SUS{POS,PIT,YAW,SIDE}_IN1_DQ channels.  I did this by modifying the activateDQ.py script to always turn them on [0].  They should now always be activated after the activateDQ script is run.

[0] This script now lives in the cds_user_apps repo at cds/c1/scripts/activateDQ.py

I updated the suspension model (/cvs/cds/rtcds/userapps/trunk/sus/c1/models/lib/sus_single_control_new.dml) to add a 5th row in the input matrix so that we can put in the calculated NULL stream vector (also have been called as Butterfly mode or Pringle mode in the past). The output of this row would go through a filter module and then is sent to the testpoint named 'C1:SUS-OPT_SENSOR_NULL' where OPT is the optic acronym. This channel is acquired in frames at 256 Hz and would be available as C1:SUS-OPT_SENSOR_NULL_DQ. After the update in the model, I built, installed and restarted models c1sus, c1mcs, c1scx, c1scy, c1su2, and c1su3. Then I restarted daqd, and everything came up nicely. After but restore, I added the null stream vector for the optics it was already known from a free swing test. ITMX, ITMY, ETMX, PRM, and SRM null stream vectors needs to be calculated from the other 4 rows. It is set to zero right now. Medm screen for the input matrix was also updated to allow seeing this row.

With the DRMI locked, I drove a line in MICH using the sensing matrix infrastructure. Then I looked at the error points of MICH, PRCL and SRCL. Initially, the sensing line oscillator output matrix for MICH was set to drive only the BS. Subsequently, I changed the --> PRM and --> SRM matrix elements until the line height in the PRCL and SRCL error signals was minimized (i.e. the change to PRCL and SRCL due to the BS moving, which is a geometric effect, is cancelled by applying the opposite actuation to the PRM/SRM respectively. Then I transferred these to the LSC output matrix (old numbers in brackets).

MICH--> PRM = -0.335 (-0.2655)

MICH--> SRM = -0.35 (+0.25)

I then measured the loop TFs - all 3 loops had UGFs around 100 Hz, coinciding with the peaks of the phase bubbles. I also ran some sensing lines and did a sensing matrix measurement, Attachment #1 - looks similar to what I have obtained in the past, although the relative angles between the DoFs makes no sense to me. I guess the AS55 demod phase can be tuned up a bit.

The demodulation was done offline - I mixed the time series of the actuator and sensor signals with a "local oscillator" cosine wave - but instead of using the entire 5 minute time series and low-passing the mixer output, I divvied up the data into 5 second chunks, windowed with a Tukey window, and have plotted the mean value of the resulting mixer output.

Unrelated to this work: I re-aligned the PMC on the PSL table, mostly in Pitch.

The "source" output of the SR785 has a DC offset of -6.66 V. I couldn't make this up.

Upshot is, this SR785 is basically not usable for TF measurements. I was using the unit to characterize the newly stuffed ISC whitening board. The initial set of measurements were sensible, and at some point, I started getting garbage data. Unclear what the cause of this is. AFAIK, we don't have any knob to tune the offset - adjusting the "offset" in the source menu, I can change the level of the offset, but only by ~1 V even if I apply an offset of 10 V. I also tried connecting the ground connection on the rear of the SR785 to the bench power supply ground, no change.

Do we have to send this in for repair?

Grrr. Let's repair the unit. Let's get a help from Chub & Jordan.

Do you have a second unit in the lab to survive for a while?

Rolf has recently written a document describing how one should fill out an IO chassis and how the numbering works out.  This can be found in the DCC at Rolf's PCIe numbering guide (T1000523).

Basically it works out that slot 1 corresponds to PCIe number 1, but slot 2 corresponds to PCIe number 6.  And so forth.  The following table gives a quick summary.

Kiwamu made the OAF screen functional today - screenshot attached.

After this, I used the measured TF of the MC1 to MCL to filter the signals from the Wilcoxon accelerometers and feed them into the MC.

The noise at 3 Hz went down by a factor of ~3. There's a little excess created at 100 Hz. Its good to see that our intuition about feed-forward is OK.

I did all of the filter calculations by adapting the scripts that Haixing, Valera, and I got going at LLO. They're all in the mDV/extra/C1 SVN.

The Wiener code predicts much better performance from using more than just 2 horizontal accelerometers, but I was too lazy to do more channels today.

I also added the Rai box to the Ranger readout today - the noise at 0.1 Hz went down by a factor of 10 and the noise at 1 Hz is close to 10^-11 m/rHz.

The Rai box ran out of batteries a couple of days ago and so the data is no good. I've put the Ranger back on the SR560 for now (but with the damping resistor removed, so the gain is 2x more than before).

As described in elog 2063 and the mevans document, we need to measure the TF of the OAF's plant, to find the delay.

At DC, the phase is 2.5deg, and at 32Hz, the delay is -4.6Hz (extrapolated from the points at ~30deg and ~38deg).  The DTT file is in /users/Templates/OAF/OAF-MCL-Delay-9Nov2011.xml . 

This gives a phase lag of 7.1deg at the Nyquist freq.

7.1 / 180 * 32 = 1.26, so ~1 cycle delay.  Not so much.  The new ADCs are waaaay faster than the old 110Bs.  Yay!

After a lot of headache, I got the OAF working again - read on for details.....................

Sometime last week, Jenne, Kiwamu, and I tried to update the OAF model to include the IIR "feed-around" path.

This path is in parallel to the existing FIR-based adaptive FXLMS stuff that Matt put in earlier. The reason for the

new path is that we want to try emulating the same FF technology which has been successful lately at LLO.

However, we were unable to make the ASS work after this work. Mostly, the build stuff worked fine, but we couldn't get DTT

to make a transfer function. The excitation channels could be selected and the excitation would actually start and get all the

way into MC1, but DTT would just hang on the first swep-sine measurement with no time-out error. Clearly our ASS building

documentation is no good. We tried using the instructions that Koji gave us for AAA, but that didn't completely work.

In particular, the 'make-uninstall-daq-ass' command gave this command:

[controls@c1ass advLigo]$ make uninstall-daq-ass
grep: target/assepics/assepics*.cmd: No such file or directory
Please make ass first
make: *** [uninstall-daq-ass] Error 1

re-arranging the order to do 'make-ass' first fixes this issue and so I have fixed this in the OAF Wiki.

The there's the whole issue with the tpchn_C3.par file. This contains all the test point definitions for the ASS/OAF machine. The main

IFO numbers are all in tpchn_C1.par and the OMC is all in tpchn_C2.par. When we do the usual build, in the 'make install-daq-ass' part:

[controls@c1ass advLigo]$ make install-daq-ass
Installing GDS node 3 configuration file
/cvs/cds/caltech/target/gds/param/tpchn_C3.par
Updating DAQ configuration file
/cvs/cds/caltech/chans/daq/C1ASS.ini
we get this .par file installed in the target area. The ACTUAL param file seems to actually be in /cvs/cds/caltech/gds/param  !!

Of course, it still doesn't work. That's because the standard build likes to point to /cvs/cds/caltech/gds/bin/awgtpman and the one that runs on

linux is actually /opt/gds/awgtpman. So I've now made a file called startup_ass.cmd.good which runs the correct one. However, the default build

will try to start the wrong one and we have to fix the 'startass' script to point to the correct one on each build. Running the correct awgtpman

allows us to get the TP data using tools like tdsdata, so far no luck with DTT.\

UPDATE (23:33): It turns out that it was my old nemesis, NTPD. c1ass had a /etc/ntp.conf file that was pointing at an ntp server called rana113. I

am not an NTP server; I don't even know what time it is. I have fixed the ntp.conf file by making it the same ass c1omc (it now points to nodus). After

this I set the date and time manually (sudo date -s "20 DEC 2009 23:27:45") and then restarted NTPD. It should now be fine even when

we reboot c1ass.

After all of this nonsense, I am able to get TP data from c1ass and take transfer functions between it and the rest of the world !

This allowed measuring the MC1 -> MCL TF finally. Its mostly flat. Data saved as Templates/OAF/OAF-MCLTF.xml

 What does OAF stand for? The entry doesn't say that. Also the acronym is not in the abbreviation page of the wiki.

Can anyone please explain that?

I fit the MC1 -> MCL TF using vectfit4.m (from mDV). The wrapper file is mDV/extra/C1/ fitMC12MCL.m.

Plotted here are the data (RED), the fit (BLUE), and the residual x10 (GREEN).

For the magnitude plot, residual is defined as ------   res = 1 - fit / data

For the phase plot the residual is defined as -------    res =  phase(data)-phase(fit)

You can see that the agreement is very good. The phase match is better than 5 deg everywhere below 10 Hz.

This TF is so smooth that we could have probably done without using this, but its good to excercise the method anyway.

At 0.1-1.0Hz, there is some coherence between MC_L and RANGER_Y & GUR_Y, see the first figure. Also GUR_Z has low noise there. So I used all five of them, increased the gains of GUR_Z from 0.1 to 0.5. Some improvement near 0.5Hz. We possibly can not do any better with these PEM measurement, as the coherence of the adapted error signal and the PEM channels is almost zero, see the second figure. May be we need to think about placing the seismometers at different places/orientations.

However, there is lot more scope at higher frequencies, lot of coherence at 5-100Hz.

There is lot of coherence between the error signal and PEM channels at 5-100Hz. We had been applying a 1Hz low pass filter to all the GUR and RANGER channels for stability. I turned those off and OAF still works with mu=0.0025, this will give us some more freedom. Kind of annoying for testing though, it takes about 45min to adapt!

In any case, there is no significant improvement at high frequencies as compared to our usual OAF performance. Also, the low frequency improvement (see previous e-log) is lost in this set up. I think, we have to adjust the number of taps and channels to do better at high frequencies. Also, delay can be important at these frequencies, needs some testing.

Did some more test to get better performance at higher frequencies.

Increased # taps to 4000 and reduced downsampling to 4, without changing the AA32 filters, from CORR, EMPH and the matching ADPT channels. But for testing I turned off AA32 from the input PEM channels. So that high frequency still gets blocked at CORR, but the adaptive filters have access to higher frequencies. Once we fix some reasonable downsampling, we should create corresponding AA filters.

I used only two channels, RANGER/GUR2_Y and GUR1_Z, and basically they had only one filter 0.1:0

This set up gave little better performance (more reduction at more frequencies), at some point even the 16HZ peak was reduced by a factor of 3. The 24Hz peak was a bit unstable, but became stable after I removed the Notch24 filters from PEM channels, to ensure that OAF is aware of those lines. There was some improvement also at the 24Hz peak.

After reducing the digital noise I did offline Wiener filtering to see how good should be online filter. I looked at the MC_F and GUR1_X and GUR1_Y signals. Here are the results of the filtering. The coherence is plotted for MC_F and GUR1_X signals.

We can see the psd reduction of the MC_F below 1 Hz and at resonances. Below 0.3 Hz some other noises are present in the MC_F. Probably tilt becomes important here.

OAF is ready to be tested. I added AA and AI filters and also a highpass filter at 0.1 Hz. OAF workes, MC stays at lock. I looked at the psd of MC_F and filter output. They are comparable, filter output adapts for MC_F in ~10 minutes but MC_F does not go down too much. Determing the right gain I unlocked the MC, while Kiwamu was measuring something. Sorry about that. I'll continue tomorrow during the daytime.

We turned on the OAF again to make sure it works. We got it to work well with the Ranger as well as the Guralp channels. The previous problem with the ACC is that Sanjit and Matt were using the "X" channels which are aligned the "Y" arm. Another casualty of our ridiculous and nonsensical coordinate system. Long live the Right Hand Rule!!

The changes that were made are:

• use of RANGER channel (with ACC_MC1_X and/or ACC_MC2_X)
• mu = 0.01, tau = 1.0e-6, ntaps = 2000, nDown = 16
• nDelay = 5 and nDelay = 7 both work (may not be so sensitive on delay at low frequencies)
• Main changes: filter bank on the PEM channels - ASS_TOP_PEM_## filters: 0.1:0, 1:, Notch24, AA32, gain 1
• Added the AI800 filter for upsampling in MC1 (should not matter)

Other parameters which were kept at usual setting:

• CORR: AI32, gain = 1
• EMPH: 0.001:0, AA32, gain = 1
• ERR_MCL: no filters, gain = 1
• SUS_MC1: no filter, gain = 1
• PEM Matrix: All zero except: (24,1), (15,2), (18,3)
• ADAPT path filter: union of CORR and EMPH filters, gain 1
• XYCOM switches # 16-19 (last four on the right) OFF 

Screenshots are attached.

Burt snapshot is kept as: /cvs/cds/caltech/scripts/OAF/snaps/ass_burt_100126_211330.snap

taken using the script: /cvs/cds/caltech/scripts/OAF/saveOAF

we should put this in ASS screen.

ERROR Detected in filter ASS_TOP_PEM_24 (RANGER): 1: was actually typed as a 1Hz high pass filter!

(Correcting this one seems to spoil the adaptation)

Possibly this makes sense, we may not want to block witness signals in the 0.1-20 Hz range.

  11:40 PM: Leaving the lab with the OAF running on 5 PEM channels (Ranger + Guralp 1&2  Y & Z). There's a terminal open on op440m which will disable the OAF in ~2.8 hours. Feel free to disable sooner if you need the MC/IFO.

  I tried downsampling value 32 (instead of 16), to see if it has any effect on OAF. Last week I encountered some stability issue - adaptation started to work, but the mode cleaner was suddenly unlocked, it could be due to some other effect too.

One point to note is that different downsampling did not have any effect on the CPU meter (I tried clicking the "RESET" button few times, but no change).

 I tried some combination of PEM channels and filters to improve OAF performance at other frequencies, where we do not have any improvement so far. There is progress, but still no success.

Here are the main things I tried:

For the ACC channels replaced the 0.1 Hz high pass filters by 3Hz high pass and turned off the 1: filter.

Then I tried to incorporate the Z ACC/GUR channels, with some reasonable combination of the others.

The Z axis Guralp and Accelerometers were making OAF unstable, so I put a 0.1 gain in all four of those.

Following the PEM  noise curves Rana has put up, we should probably use

• two ACC_Y channels (3:0, Notch24, AA32)
• two GUR_Z channels (filters: 0.1:0, 1:, AA32, gain 0,1)
• one RANGER_Y, just because it works (0.1:0, 1:, Notch24, AA32)

In the end I tried this combination, it was stable after I reduced the GUR_Z gain, but looked very similar to what we got before, no improvement at 5Hz or 0.5Hz. But there was a stable hint of better performance at > 40Hz.

Possibly we need to increase the GUR_Z gain (but not 1) and try to use ACC_Z channels also. Since we can not handle many channels, possibly using one GUR_Z and one ACC_Z would be worth checking.

I am leaving the OAF disabled, so there should be nothing that goes to the suspensions from the OAF. 

Disabled for the OAF means all the outputs are multiplied by 0 just before the signals are sent back over to the LSC system to be summed in and sent to the suspensions.  So 0 times anything should mean that the OAF isn't injecting signals.

In other news, while Mirko was trying to figure out the c-code, I began making a new OAF screen.  It is still in progress, but it's in c1oaf/master/C1OAF_OVERVIEW.adl  If you open it up, you can see for yourself that the OAF is disabled.  (Eventually I'll put an enable/disable button on the LSC screen also, but that hasn't happened yet.)

No leaving the OAF running until you're sure (sure-sure, not kind of sure, not pretty sure, but I've enabled guardians to make sure nothing bad can happen, and I've been sitting here watching it for 24+ hours and it is fine) that it works okay.

OAF (both adaptive and static paths) were left enabled, which was kicking MC2 a lot.  Not quite enough that the watchdog tripped, but close.  The LSCPOS output for MC2 was in the 10's or 100's of thousands of counts.  Not okay.

This brings up the point though, which I hadn't actively thought through before, that we need an OAF watchdog.  An OAF ogre?  But a benevolent ogre.  If the OAF gets out of control, it's output should be shut off.  Eventually we can make it more sophisticated, so that it resets the adaptive filter, and lets things start over, or something. 

But until we have a reliable OAF ogre, no leaving the adaptive path enabled if you're not sitting in the control room.  The static path should be fine, since it can't get out of control on it's own.

Especially no leaving things like this enabled without a "I'm leaving this enabled, I'll be back in a few hours to check on it" elog! 

I am meditating on the OAF, and had it running and calculating things.  I had the outputs disabled so I could take reference traces in DTT, but the Adapt block was calculating for MCL.  At some point, all the numbers froze, and the CPU meter had gone up to ~256ms.  Usually it's around ~70 or so for the configuration I had (2 witness sensors and one degree of freedom enabled....no c-code calculations on any other signals).  The "alive" heartbeat was also frozen.

I ssh'ed into c1lsc, ran ./startc1oaf in the scripts directory, and it came back just fine.

Anyhow, I don't know why it got funny, but I wanted to record the event for posterity.  I'm back to OAFing now.

Den noticed this, and will write more later, I just wanted to sum up what Alex said / did while he was here a few minutes ago....

Errors are probably really happening.... c1oaf computer status 4-bit thing:  GRGG.  The Red bit is indicating receiving errors.  Probably the oaf model is doing a sample-and-hold thing, sampling every time (~1 or 2 times per sec) it gets a successful receive, and then holding that value until it gets another successful receive. 

Den is adding EPICS channels to record the ERR out of the PCIE dolphin memory CDS_PART, so that we can see what the error is, not just that one happened.

Alex restarted oaf model:  sudo rmmod c1oaf.ko, sudo insmod c1oaf.ko .  Clicked "diag reset" on oaf cds screen several times, nothing changed.  Restarted c1oaf again, same rmmod, insmod commands.

Den, Alex and I went into the IFO room, and looked at the LSC computer, SUS computer, SUS I/O chassis, LSC I/O chassis and the dolphin switch that is on the back of the rack, behind the SUS IO chassis.  All were blinking happily, none showed symptoms of errors.

Alex restarted the IOP process:  sudo rmmod c1x04, sudo insmod c1x04.  Chans on dataviewer still bad, so this didn't help, i.e. it wasn't just a synchronization problem.  oaf status: RRGG. lsc status: RGGG. ass status: RGGG.

sudo insmod c1lsc.ko, sudo insmod c1ass.ko, sudo insmod c1oaf.ko .  oaf status: GRGG. lsc status: GGGG. ass status: GGGG.  This means probably lsc needs to send something to oaf, so that works now that lsc is restarted, although oaf still not receiving happily.

Alex left to go talk to Rolf again, because he's still confused.

Comment, while writing elog later:  c1rfm status is RRRG, c1sus status is RRGG, c1oaf status is GRGG, both c1scy and c1scx are RGRG.  All others are GGGG.