[root@c1omc controls]# /opt/gds/awgtpman -2 &
[1] 16618
[root@c1omc controls]# mmapped address is 0x55577000
32 kHz system
Spawn testpoint manager
no test point service registered
Test point manager startup failed; -1

[1]+  Exit 1                  /opt/gds/awgtpman -2

This turned out to be fallout from the /cvs/cds transition.  Remounting and restarting fixed it.

rob, koji, steve

We noticed some water (about a cup) on the floor under the NESLAB chiller today.  We put the chiller up on blocks and took off the side panel for a cursory inspection, but found no obvious leaks.  We'll keep an eye on it.

A cosmic ray struck the RFM in the framebuilder this afternoon, causing hours of consternation.  The whole FE system is just now coming back up, and it appears the mode cleaner is not coming back to the same place (alignment).

rob, jenne

I measured the mode cleaner open loop gain with the HP3563A. 

The UGF is 64kHz, phase margin is 28 deg.

The PSL/IOO combo has not been behaving responsibly recently. 

The first attachment is a 15 day trend of the MZ REFL, ISS INMON, and MC REFL power.  These show two separate problems--recurring MZ flakiness, which may actually be a loose cable somewhere which makes the servo disengage.  Such disengagement is not as obvious with the MZ as it is with other systems, because the MZ is relatively stable on its own.  The second problem is more recent, just starting in the last few days.  The MC is drifting off the fringe, either in alignment, length, or both.  This is unacceptable.

The second attachment is a two-day trend of the MC REFL power.  Last night I carefully put the beam on the center of the MC-WFS quads.  This appears to have lessened the problem, but it has not eliminated it. 

It's probably worth trying to re-measure the MCWFS system to make sure the control matrix is not degenerate. 

Mode cleaner length measured tonight.

33196198

132784792

165980990

199177188

[Tag by KA: modulation frequency, MC length]

This happened when I plugged the cards into a crate with computers, which apparently is a no-no.  The extender cards only go in VME crates full of in-house, LIGO-designed electronics.

We had a redo of elog entry 975 tonight.  The noisy OSEM was fixed by jiggling the rack end of the long cable.  Don't know exactly where--I also poked around the OSEM PD interface board.

In the attached PDF the reference trace is the noisy one.

much grief.  somehow a burt restore of c1iscepics failed to work, and so the LSC XYCOM settings were not correct.  This meant that the LSC whitening filter states were not being correctly set and reported, making it difficult to lock for at least the last week or so.

It's been a miserable week for lock acquisition, with each night worst than the last.  The nadir was around Sunday night, when I couldn't even get a PRM to lock stably, which meant that the auto-alignment scripts could not finish successfully.  It now appears that was due to some XYCOM mis-settings. 

We've also been having problems with timing for c1susvme2.  Attached is a one-hour plot of timing data for this cpu, known as SRM.  Each spike is an instance of lateness, and a potential cause of lock loss.  This has been going on for a quite a while.

Tonight we also encountered a large peak in the frequency noise around 485 Hz.  Changing the MZ lock point (the spot in the PZT range) solved this.

Good progress in IFO locking tonight, with the arm powers reaching about half the full resonant maximum. 

Still to do is check out some weirdness with the OMC DAC, fix the wireless network, and look at c1susvme2 timing.

 I've watchdogged all the suspensions while I mess around with computers.  If no one else is using the IFO, we can leave them undamped for a couple of hours to check the resonant frequencies, as long as I don't interrupt data streams with my computer hatcheting.

Attached is a 3 day trend of SRM CPU timing info.  It clearly gets better (though still problematic) at some point, but I don't know why as it doesn't correspond with any work done.  I've labeled a reboot, which was done to try to clear out the timing issues.  It can also be seen that it gets worse during locking work, but maybe that's a coincidence.

More progress with locking tonight, with initial acquisition and power ramps working.  The final handoff to RF CARM still needs work.

I found the wireless router was unplugged from the network--just plugging in the cable solved the problem.  For some reason that RJ45 connector doesn't actually latch, so the cable is prone to slipping out of the jack.

It got worse again, starting with locking last night, but it has not recovered.  Attached is a 3-day trend of SRM cpu load showing the good spell.

I took the output of the OMC DAC and plugged it directly into an OMC ADC channel to see if I could isolate the OMC DAC weirdness I'd been seeing.  It looks like it may have something to do with DTT specifically.

Attachment 1 is a DTT transfer function of a BNC cable and some connectors (plus of course the AI and AA filters in the OMC system).  It looks like this on both linux and solaris.

Attachment 2 is a transfer function using sweepTDS (in mDV), which uses TDS tools as the driver for interfacing with testpoints and DAQ channels. 

Attachment 3 is a triggered time series, taken with DTT, of the same channels as used in the transfer functions, during a transfer function.  I think this shows that the problem lies not with awg or tpman, but with how DTT is computing transfer functions. 

I've tried soft reboots of the c1omc, which didn't work.   Since the TDS version appears to work, I suspect the problem may actually be with DTT.

quake coming through.  I've re-enabled optic damping (except ETMY), and left off the oplevs for now.  We can do a resonant-f  check over the weekend.

Looks like it was a magnitude 5 near Olancha, where they sell really good fresh jerky.  quake

Earthquake Details

• This event has been reviewed by a seismologist.

latest news: there's actually been about a dozen earthquakes in Keeler in the last couple hours:   http://earthquake.usgs.gov/eqcenter/recenteqsus/Maps/special/California_Nevada_eqs.php

-Rana

Late last night after the ETMY settled down from the quake I made some more progress in locking, with the handoff to RF CARM succeeding once.  The final reduction of the CARM offset to zero didn't work, however.

Working well tonight: the handoff of CARM to RF (REFL2I), successful reduction of CARM offset to zero, and transition control of MCL path to the OUT1 from the common mode board.  All that's left in lock acquisition is to try and get the common mode bandwidth up and the boost on.

Lock acquisition working well tonight.  Was able to engage CM boost (not superboost) with bandwidth of ~10kHz.  Also succeeded once in handing off DARM to DC readout.

 Last week, Alex recompiled the c1susvme2 code without the decimation filters for the OUT16 channels, so these channels are now as aliased as the rest of them.  This appears to have helped with the timing issues: although it's not completely cured it is much better.  Attached is a five day trend.

Last night, 2+ hour lock, probably broken by me driving too hard (DARM_EXC).

The IFO can now be locked during the daytime.  Well, it's locked now.

I pushed the "closed loop" button on PZT2 YAW around 3:40 pm today, then roughly recentered it using the DC Offset knob on the PiezoJena controller and the IP ANG QPD readbacks.  There was a large DC shift.    We'll watch and see how much it drifts in this state.

The Chameleon HB (by Knox) video switch that we use for routing video signals into the control room monitors is broken.  Well, either it's broken, or something is wrong with the mv162 EPICS IOC which communicates with it via RS-232.  Multiple reboots/resets of both machines has not yet worked.  The CHHB has two RS-232 inputs--I switched to the second one, and there is now one signal coming through to a monitor but no switching yet. I've been unable to further debug it because we don't have anything in the lab (other than the omega iserver formerly used for the RGA logger) which can communicate with RS-232 ports.  I've been trying to get this thing (the iserver) working again, but can't communicate with it yet.  For now I'm just going to bypass the video switch entirely and use up all the BNC barrel connectors in the lab, so we can at least have the useful video displays back.

the mode cleaner was having trouble locking in a 00 mode, needing several tries.  I changed the MC2 coil biases, and it seems better for now.

The 33MHz mod depth is now controlled by the OMC (C1:OMC-SPARE_DAC_CH_15).  The setting to give us the same modulation depth as before is 14000 (in the offset field).

So, it appears that one doesn't even have to change the Marconi set frequency to alter the phase of the output signal.  It appears that other front panel actions (turning external modulations on/off, changing the modulation type) can do it as well.  At least that's what I conclude from earlier this morning, when after setting up the f2 Marconi (166MHz) for external AM, the double-demod handoff in the DRMI no longer worked.  Luckily this isn't a real problem now that we have the setDDphases and senseDRM scripts. 

Lock acquisition has gone bad tonight. 

The initial stage works fine, up through handing off control of CARM to MCL.  However, when increasing the AO path (analog gain), there are large DC shifts in the C1:IOO-MC_F signal.  Eventually this causes the pockels cell in the FSS loop to saturate, and lock is lost. 

I measured the mode cleaner open loop gain.  It's around 60kHz with 29 degs of phase margin.

A 30-day trend of the PCDRIVE from the FSS.

 This problem has disappeared.  I don't know what it was. 

The first plot shows one of the symptoms.  The second plot is a similar section taken from a more normal acquisition sequence the night before.

All is not perfect, however, as now the handoff to RF CARM is not working.

LockAcq is back on track, with the full script working well.  Measurements in progress.

The problem has returned.  I still don't know what it is, but it's making me angry. 

I never actually figured out exactly what was wrong in entry 2162, but I managed to circumvent by changing the time sequence of events in the up script, moving the big gain increases in the common mode servo to the end of the script.  So the IFO can be locked again.

It won't start--it just sits at Waiting for EPICS BURT, even though the EPICS is running and BURTed.

[controls@c1omc c1omc]$ sudo ./omcfe.rtl
cpu clock 2388127
Initializing PCI Modules
3 PCI cards found
***************************************************************************
1 ADC cards found
        ADC 0 is a GSC_16AI64SSA module
                Channels = 64
                Firmware Rev = 3

***************************************************************************
1 DAC cards found
        DAC 0 is a GSC_16AO16 module
                Channels = 16
                Filters = None
                Output Type = Differential
                Firmware Rev = 1

***************************************************************************
0 DIO cards found
***************************************************************************
1 RFM cards found
        RFM 160 is a VMIC_5565 module with Node ID 130
***************************************************************************
Initializing space for daqLib buffers
Initializing Network
Waiting for EPICS BURT




From looking at the recorded data, it looks like the c1omc started going funny on the afternoon of Nov 5th, perhaps as a side-effect of the Megatron hijinks last week.

It works when megatron is shutdown.

Had been disconnected for about two weeks.  I found a partially seated 4-pin LEMO cable coming from the OSEM PD interface board. 

I've changed the watchdog rampdown script so it brings the SUS watchdogs to 220, instead of the 150 it previously targeted.  This is to make tripping less likely with the jackhammering going on next door.  I've also turned off all the oplev damping.

Locking has gone sour.  The CARM to MCL handoff, which is fairly early in the full procedure and usally robust, is failing reliably. 

As soon as the SUS-MC2_MCL gain is reduced, lock is broken.  There appears to be an instability around 10Hz.  Not sure if it's related.

 Five day minute trend.  FAST_F doesn't appear to have gone crazy.

 Whatever the locking problem was, the power of magical thinking has forced it to retreat for now.  The IFO is currently locked, having completed the full up script.  One more thing for which to be thankful.

Came in, found all front-ends down.

Keyed a bunch of crates, no luck:

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS 

Powered off/restarted c1dcuepics.  Still no luck.

Powered off megatron.  Success!  Ok, maybe it wasn't megatron.  I also did c1susvme1 and c1susvme2 at this time.

BURT restored to Nov 26, 8:00am

But everything is still red on the C0_DAQ_RFMNETWORK.adl screen, even though the front-ends are running and synced with the LSC.  I think this means the framebuilder or the DAQ controller is the one in trouble--I keyed the crates with DAQCTRL and DAQAWG a couple of times, with no luck, so it's probably fb40m.    I'm leaving it this way--we can deal with it tomorrow.

 A "Data Receiving Error" usually indicates a problem with the framebuilder/testpoint manager, rather than the front-end in question.  I'd bet there's a DTT somewhere that's gone rogue.

Taking  a cue from entry 2346, I immediately went for the nuclear option and powered off fb40m.  Someone will probably need to restart the backup script.

There's a large broadband increase in the MC_F spectrum.  I'm not totally sure it's real--it could be some weird bit-swapping thing.  I've tried soft reboots of c1susvme2 and c1iovme, which haven't helped.  In any case, it seems like this is preventing any locking success today.  Last night it was fine.

 Rebooting c1iovme (by keying off the crate, waiting 30 seconds, and then keying it back on and restarting) has resolved this.  The frequency noise is back to the 'usual' trace.