I had a go at trying to bring c1susvme1 back online.  The first few times I hit the physical reset button, I saw the same error that Joe mentioned, about needing to check some cables.  I tried one round of rebooting c1sosvme, c1susvme2 and c1susvme1, with no success.  After a few iterations of jiggle cables/reset button/ctrl-x on c1susvme1, it came back.  I ran the startup.cmd script, and re-enabled the suspensions, and Mode Cleaner is now locked.  So, all systems are back online, and I'm crossing my fingers and toes that they stay that way, at least for a little while.

Power cycling c1dcuepics seems to have fixed the EPICs channel problems, and c1lsc, c1asc, and c1iovme are talking again.

I burt restored c1iscepics and c1Iosepics from the snapshot at 6 am this morning.

However, c1susvme1 never came back after the last power cycle of its crate that it shared with c1susvme2.  I connected a monitor and keyboard per the reboot instructions.  I hit ctrl-x, and it proceeded to boot, however, it displays that there's a media error, PXE-E61, suggests testing the cable, and only offers an option to reboot.  From a cursory inspection of the front, the cables seem to look okay.  Also, this machine had eventually come back after the first power cycle and I'm pretty sure no cables were moved in between.

At around 9:45 the RFM/FB network alarm went off, and I found c1asc, c1lsc, and c1iovme not responding. 

I went out to hard restart them, and also c1susvme1 and c1susvme2 after Jenne suggested that.

c1lsc seemed to have a promising come back initially, but not really.  I was able to ssh in and run the start command.  The green light under c1asc on the RFMNETWORK status page lit, but the reset and CPU usage information is still white, as if its not connected.   If I try to load an LSC channel, say like PD5_DC monitor, as a testpoint in DTT it works fine, but the 16 Hz monitor version for EPICs is dead.  The fact that we were able to ssh into it means the network is working at least somewhat.

I had to reboot c1asc multiple times (3 times total), waiting a full minute on the last power cycle, before being able to telnet in.  Once I was able to get in, I restarted the startup.cmd, which did set the DAQ-STATUS to green for c1asc, but its having the same lack of communication as c1lsc with EPICs.

c1iovme was rebooted, was able to telnet in, and started the startup.cmd.  The status light went green, but still no epics updates.

The crate containing c1susvme1 and c1susvme2 was power cycled.  We were able to ssh into c1susvme1 and restart it, and it came back fully.  Status light, cpu load and channels working.  However I c1susvme2 was still having problems, so I power cycled the crate again.  This time c1susvme2 came back, status light lit green, and its channels started updating.

At this point, lacking any better ideas, I'm going to do a full reboot, cycling c1dcuepics and proceeding through the restart procedures.

[Rana, Jenne]

There is some craziness going on with the delay in the PEM path for the OAF.  We plot the difference between the C1:PEM-SEIS_GUR1_X and C1:ASS-TOP_PEM_10.  These are physically the same channel, plugged into the PEM ADCU, and then the signal is used as a regular PEM channel, and is also sent to the ASS computer and used there for the OAF system.  As you can see in the blue trace on the bottom plot, there is a huge amount of delay, and it's very noisy.  We also plot the _GUR2_X / ASS-TOP_PEM_2 pair (red), and it has a similar amount of delay, but it is not nearly as fuzzy and noisy.  For comparison, we plot the SUS-MC2_MCL (which is identical to IOO-MC_L) and ASS-TOP_ERR_MCL pair (green), and they don't have any big overall delay problems, so it's not totally a problem with the signals getting to the ASS computer.

This problem was present during/after all of the following attempts to fix it:

* The sample rate on the ASS computer is 2048.  The PEM channels were being acquired the ADCU at 512.  We changed the ADCU sampling rate to 2048 to match.

* We soft rebooted the ASS computer, in case it was a timing problem.

* Doing a "sudo shutdown -r now" while logged in as controls.

We might also try resetting/power cycling c0dcu in the morning.  Alex has been emailed to help us try to figure this out.

In other news, the time delay that we measure from the plot gives us 180degrees in ~210Hz.  This corresponds to a little more than 2msec of delay, with the C1:ASS version lagging behind the C1:PEM version.  (2 samples at 840Hz) Converting to the 2048 sampling rate, we have a delay of 4.8, so 5 front-end cycles.  Since Rana measured this morning that the delay indicated by the transfer function is 10 cycles, and this delay shows that the ASS lags the actual seismometer signal by 5 cycles, we should subtract this 5 from the 10 from the transfer function, giving us a final sample-and-hold delay of 5.  Coincidentally(?), 5 is the delay that was found in the C1:ASS-TOP screen, after it's one year of dormancy.  The point of the delay feature in the code is to help match the delay in the two signal paths: the PEM path and the output path of the filter.  Since the output has a lag of 10, and the PEM path has a lag of 5, to make them match, we artificially put in a delay of 5.

 I spiffed up the order of the cables / sensors plugged into the PEM ADCU.  Now all of the seismometers are labeled as Rana left them, and the 2 Guralp's have their sets of 3 channels next to eachother in channel-number-land.  None of the accelerometer names/cabling have changed recently.  In the table, Cable-label refers to the physical tag tied to the end of the cables plugged into the ADCU...they are meant to be descriptive of what seismometer channels they are hooked up to, and then the names change to something useful for us when they come into the DAQ system.  Also, the labels of input channels on the ASS_TOP_PEM screen have been updated accordingly.

I took the pictures of all racks of electronics yesterday, and then uploaded these pictures on the wiki.

http://lhocds.ligo-wa.caltech.edu:8000/40m/Electronics

You can see them by clicking "pictures" in the wiki page.

I remeasured the OAF time delay using the OAF-TF template from the Templates/ directory.

Troublingly, I found the MC1 dewhitening switches set OFF - please make sure that the MC1 dewhitening is back ON after each OAF tuning so that the interferometer locking is not hosed.

The OAF-TF template had the excitation amplitude set ~20x too high. I reduced it and the coherence was still > 0.95. The phase at 32 Hz was still ~126 deg as Jenne had measured, but since the phase at DC is 180 deg, the overall phase lag is just 180-126 = 54 deg. So the delay should be 54/180 * 32 = 9.7 => 10. Luckily, Jenne is working on an instructional manual for OAF that will make all of this crystal clear.

I drove MC2 in POS and used the resulting response in MC_F to calibrate the IOO-MC_L channel.

Yoichi did an excellent job of calibrating MC_F last year. I have used his calibration of MC_F (220 Hz/count @ DC) to get the MC_L calibration at DC as well as at high frequencies. The hardware dewhitening was OFF for all these measurements.

Method

1. For the DC measurement I excited C1:SUS-MC2_MCL_EXC at 0.0731 Hz. At these frequencies, the MC_L path has much more gain than the MC_F path. So this excitation at the error point makes the length path to drive itself to cancel the digital excitation. Since the overall MC servo gain is huge, this causes the MC_F path to compensate the residual MC_L motion. One can simply take the ratio of MC_L/MC_F to get the calibration of MC_L in Hz.

2. For the AC measurement, I engaged FM9 of the MC2_MCL filter bank. This guy is an elliptic LP with a notch at 660.38 Hz. I then drove MC2_LSC at 660.38 Hz with a sine wave of 500 counts amplitude. The notch makes the gain of the MC_L feedback zero at that frequency. So MC_F has to do all the work. We can simply measure the ratio of MC2_LSC/MC_F to get the AC calibration of MC2_MCL_OUT (aka IOO-MC_L) and MC2_LSC_OUT (aka LSC-MC_L).

Results:

MCF/MCL @ 0.0731 Hz = 569.4. So the MC_L calibration at DC is 220 x 569.4 = 125 kHz/count or 6.02 nm/count.

From this we would expect the AC calibration to be (6 nm/count)*(660.38/f_pend)^2 = 13.0 x10^-15 m/count.

The AC measurement gave 1445 counts_peak** of MC_F for the 500 counts (peak) excitation in MC2_LSC. From Yoichi's entry we get that the high frequency calibration of MC_F should be 0.089 Hz/count. So the MC_L calibration at 660 Hz is 0.089*1445/500 = 0.25 Hz / count or 12.3 x 10^-15 m/count. So the AC/DC ratio is close to 1.

Splitting the difference, the new official MC_L calibration is 5.87 nm/counts @ DC with a complex pole pair at 0.972 Hz.

** note:  To convert from the peak height observed in DTT with a 50% Overlap Hanning window you must use the following intuitive formula:  counts_peak = (counts / rHz) * sqrt(2 * BW). In this case, BW is the number that DTT reports as BW on the screen, NOT the BW that you asked for on the measurement tab.

*** note: when measuring peak heights in a DTT FFT, make sure to unclick the box for 'Bin' under the config tab. Bin suppresses peaks in a plot with a lot of points and is ON by default.

**** note: I have updated the MCF reference in the Templates directory with the Yoichi calibration - spectrum attached. This is probably the most accurate MCF spectrum we have ever put in the elog in the history of the 40m. The implication is that the VCO phase noise is ~5 mHz/rHz. Not bad.

***** note: with the OAF off, I drove a 1.55 Hz sine wave into MC1 and measured the ratio of MC1_MCL_OUT/IOO-MC_L. This gives the DC calibration of MC1_MCL_OUT = 7.98 nm/count.

 Here's a plot of the spectra of the seismometers and MCL. The coherence shows which axes are aligned right now: MC1_X is coherent with GUR_NS which means that its mis-oriented.

I've now swapped the "MC1" cables: so the old "NS" now goes into EW and the old EW now goes into NS. VERT is unchanged.

Also fixed the channel names - the Guralp previously named MC1 is now GUR1 and the other one is GUR2. Also no more EW, NS, & VERT. Its all XYZ.

DAQD restarted with the new channel names.

Awesome.

I propose that anyone who tries to do this kind of thoroughgoing cleaning should make an e-mail to call everyone available to join just for some hours
because every member has a responsibility to keep the lab organized.

And we have the list of things to do: Electronics (now it is halfway) / Cables / Optics / Screws / Tools ...

I spent part of the afternoon cleaning up the area next to the Mode Cleaner where we keep all of our RF stuff:  Attenuators, BNC/SMA/LEMO adapters, Mini-Circuits items, and all sorts of other things which are useful while looking at our electronics/RF stuff.

We got another set of "Lyon" drawers, which aided in the organization process....Bob ordered 2, so we now have a 'spare' drawer set if anyone can think of something else to organize (unless this was premeditated for optics or something else?).

As you can see in the picture, (1) it's no longer a total disaster over there, and (2) some of the drawers have sub-divisions to make it faster and easier to find what you're looking for.  Please help out by putting things away in their proper place, and adding more labels or dividers to the drawers if there's something else which needs a 'spot'.

 As Rana pointed out to me last night, I was using continuous phase, which is not good.  When using regular phase, I find: (29.85Hz, 131.216deg), (37.06Hz, 113.963deg), so extrapolating gives (32Hz, 126.07deg).  Plugging this in to our handy-dandy formula, we get a delay of 22.4, so we should try both 22 and 23.

I'm back to terrorize the PSL table again. The pictures I took yesterday were rubbish--today I'm using a clamp that Steve was nice enough to loan me. I'm starting now, at 10:09 am.

Lock acquisition working well tonight.  Was able to engage CM boost (not superboost) with bandwidth of ~10kHz.  Also succeeded once in handing off DARM to DC readout.

I set the FSS slow actuator adj to -5.6 at the lunch time. It gave a little help at that time. Now max of the MC Trans is comming back somehow. I hope the MC Trans level is as good as before, if the HEPA is slowed down.

[Jenne, Sanjit]

As per Matt's instructions in his OAF document (elog 395) in the Tuning section, Sanjit and I took a transfer function measurement from the output of the OAF system, to the input.  i.e. we're trying to measure what happens out in the real world when we push on MC1, and how that is fed back to the input of our filter as MC_L.  The game plan is to measure this transfer function, and read off the phase at the nyquist frequency, and use this value to calculate the appropriate sample-and-hold delay to be used in the OAF.  The downsample rate for the OAF is 32, so that we're running at 64Hz instead of the 2048Hz of the front-end.  Thus, our Nyquist frequency is 32Hz.

                            DownSampleRate

Phase@Nyquist * ------------------------

                                    180

In the attached figure we do a swept sine from CORR_EXC to ERR_EMPH_OUT to determine the transfer function.  Here, we turn off all of the filters in both the CORR and EXC banks, because those are already matched/taken into account in the PEM filter banks. 

Using the cursor on DTT, we find that the phase at 29.85Hz is -228.8deg, and at 37.06Hz is -246.0deg.  Extrapolating, this means that at 32Hz, we expect about -234deg phase.  Using our handy-dandy formula, this means that we should try a delay of 41 or 42 (41.6 is between these two...) 

We'll give this a shot!

In the revival of the experiement length measurement for the recycling cavities, I turned the auxiliary NPRO back on. The shutter is closed.

I also recollected all the equipment of the experiment after that during the summer it had been scattered around the lab to be used for other purposes (Joe and Zach's cameras and Stephanie and Koji's work with the new EOM).

All done (for now). That wasn't so bad, was it?

I have been commissioned to take pictures of the PSL table so that it can be diagrammed. I am starting now (1:42 pm, 10/5/09).

The PSL output looks smaller than the incident. Try to FSS Slow actuator adj of -5.6 (nominal), instead of -3.5.

Alberto, Steve,

While I was moving a cart near by the PSL table I pushed the red emergency button that turns off the PSL laser. We had to unlock the button and then power cycle the laser driver to turn the laser back on.

I relocked MZ, FSS, PMC and I'm now waiting for the power to finish ramping up back to the previous value.

Working well tonight: the handoff of CARM to RF (REFL2I), successful reduction of CARM offset to zero, and transition control of MCL path to the OUT1 from the common mode board.  All that's left in lock acquisition is to try and get the common mode bandwidth up and the boost on.

Late last night after the ETMY settled down from the quake I made some more progress in locking, with the handoff to RF CARM succeeding once.  The final reduction of the CARM offset to zero didn't work, however.

quake coming through.  I've re-enabled optic damping (except ETMY), and left off the oplevs for now.  We can do a resonant-f  check over the weekend.

Looks like it was a magnitude 5 near Olancha, where they sell really good fresh jerky.  quake

Earthquake Details

• This event has been reviewed by a seismologist.

latest news: there's actually been about a dozen earthquakes in Keeler in the last couple hours:   http://earthquake.usgs.gov/eqcenter/recenteqsus/Maps/special/California_Nevada_eqs.php

-Rana

I took the output of the OMC DAC and plugged it directly into an OMC ADC channel to see if I could isolate the OMC DAC weirdness I'd been seeing.  It looks like it may have something to do with DTT specifically.

Attachment 1 is a DTT transfer function of a BNC cable and some connectors (plus of course the AI and AA filters in the OMC system).  It looks like this on both linux and solaris.

Attachment 2 is a transfer function using sweepTDS (in mDV), which uses TDS tools as the driver for interfacing with testpoints and DAQ channels. 

Attachment 3 is a triggered time series, taken with DTT, of the same channels as used in the transfer functions, during a transfer function.  I think this shows that the problem lies not with awg or tpman, but with how DTT is computing transfer functions. 

I've tried soft reboots of the c1omc, which didn't work.   Since the TDS version appears to work, I suspect the problem may actually be with DTT.

Here's the gist of the requirements on the 5x frequency multiplier for the upgrade (see attachemnt - despite the preview down here in the elog, they're 3 pages).

An extended version is available on the wiki.

A more complete document is under work and will soon be available on the wiki as well.

we set the offsets for the MCWFS DC and for the MCWFS demod outputs and then turned off the lights put the MZ at half fringe and then centered the spots on the MCWFS heads.

The MCREFL beam looks symmetric again and the MC REFL power is low. 

It got worse again, starting with locking last night, but it has not recovered.  Attached is a 3-day trend of SRM cpu load showing the good spell.

The attached shows the 200 day '10-minute' trend of the CPU meters and also the room temperature.

To my eye there is no correlation between the signals. Its clear that c1susvme2 (SRM LOAD) is going up and no evidence that its temperature.

More progress with locking tonight, with initial acquisition and power ramps working.  The final handoff to RF CARM still needs work.

I found the wireless router was unplugged from the network--just plugging in the cable solved the problem.  For some reason that RJ45 connector doesn't actually latch, so the cable is prone to slipping out of the jack.

Ops. I restored the damping of the suspensions at around 16:30.

MZ work has been done. I did not change anything on the circuit.

Recently we observed that the MZ PZT output was sticking at a certain voltage. I found the reason.
Shortly to say "we must return the PZT Ramp offset to 0, after the lock"

I am going to write a MZ auto lock script someday, to do it automatically.

According to the resister values used in the circuit, the PZT HV output voltage is determined by the following formula:

V_PZT = 150 - 12 V_ctrl - 24 Vramp

Here the ramp voltage Vramp moves from -10V to +10V, the feedback control voltage V_ctrl moves from -13V to +13V.
The baseline offset of 150V is provided in the circuit board.

When V_ramp is 0, V_PZT runs from 0 to 300. This is just enough for the full scale of the actual V_PZT range,
that is 0V~280V.

If any Vramp offset is given, V_PZT rails at either side of the edges. This limits the actual range of the PZT out.

This is not nice, but is what happened recently.

Attached is a 3 day trend of SRM CPU timing info.  It clearly gets better (though still problematic) at some point, but I don't know why as it doesn't correspond with any work done.  I've labeled a reboot, which was done to try to clear out the timing issues.  It can also be seen that it gets worse during locking work, but maybe that's a coincidence.

 I've watchdogged all the suspensions while I mess around with computers.  If no one else is using the IFO, we can leave them undamped for a couple of hours to check the resonant frequencies, as long as I don't interrupt data streams with my computer hatcheting.

I will investigate the MZ board. I will unlock MZ (and MC).

 Round 3 for the day of MC2 watchdogs tripping.

 The EQ did not change the input beam pointing. All back to normal, except MC2 wachdogs tripped again.

MZ stayed unlocked. Now It was relocked.

Earthquake  of magnitude 5.0  shakes ETMY loose.

MC2 lost it's damping later.

Good progress in IFO locking tonight, with the arm powers reaching about half the full resonant maximum. 

Still to do is check out some weirdness with the OMC DAC, fix the wireless network, and look at c1susvme2 timing.

[Sanjit, Jenne]

The up and down scripts accessible from the OAF (still C1:ASS-TOP) screen are now totally functional and awesome.  They are under the blue ! button.  The up script can either be for the Seismometers, or the Accelerometers at this time.  The only difference between these 2 is which burt restore file they look at:  the seismometer version puts all 7 seismometer channels in the PEM selecting matrix, while the accelerometer version puts the 6 accelerometer channels in that matrix.  Both scripts also turn on HP_1mHz filters in the ERR_EMPH filter bank and all of the witness filter banks, and the AA32 and AI32 filters in ERR_EMPH, CORR and PEM filter banks.  This makes all of the starting filters the same between the witness paths and the error path.

If you want to use a different combination of sensors, run one of the up scripts, then change the PEM matrix by hand. 

The down script disables the output to the optics, and resets the adapted filter coefficients.  DO NOT use this script if you're trying to "pause" the filter to take some nice long averages.

I'm blaming Zach on this one, for trying to upload a pic into the ATF elog (even though he claims it was small....)  Blame assigned: check.  Elog restarted: check.

It's been a miserable week for lock acquisition, with each night worst than the last.  The nadir was around Sunday night, when I couldn't even get a PRM to lock stably, which meant that the auto-alignment scripts could not finish successfully.  It now appears that was due to some XYCOM mis-settings. 

We've also been having problems with timing for c1susvme2.  Attached is a one-hour plot of timing data for this cpu, known as SRM.  Each spike is an instance of lateness, and a potential cause of lock loss.  This has been going on for a quite a while.

Tonight we also encountered a large peak in the frequency noise around 485 Hz.  Changing the MZ lock point (the spot in the PZT range) solved this.

much grief.  somehow a burt restore of c1iscepics failed to work, and so the LSC XYCOM settings were not correct.  This meant that the LSC whitening filter states were not being correctly set and reported, making it difficult to lock for at least the last week or so.

cosmic rays in cars

We had a redo of elog entry 975 tonight.  The noisy OSEM was fixed by jiggling the rack end of the long cable.  Don't know exactly where--I also poked around the OSEM PD interface board.

In the attached PDF the reference trace is the noisy one.

Probably there is the same mistake for the PMC gain slider. Possibly on the FSS slider, too???

The previous "+15" was Vctrl = 0.25 [V]. Which was +18 dB.

Since we used to run with a gain slider setting of +15 dB on the MZ, I wanted to check that the new setting of +30dB was OK. It is.

To check it I turned it up and looked for some excess noise in the ISS or in the MC. There was none. I also set the input offset slider by unlocking the PMC and zeroing the mixer monitor readback. The new slider setting is -6.5V.

I don't know why we would need more gain on the MZ loop, but we can have some if we want it by turning up the gain before the servo (optical or RF). The attached plot shows the MC_F and ISS signals with the ISS loop on and off. There was no change in either of these spectra with the MZ gain high or low.

The MZ work completed. I replaced the bad cross connection terminal. The gain slider is working now.

I looked at the error spectrum on an FFT analyzer. I could see the lock was more tight.

Then I proceeded to the MZ epics panel.

1) C1:PSL-MZ_MZTRANSPD has no meaning (not connected). So I put  C1:PSL-ISS_INMONPD as the MZ trans monitor.

2) The EPICS setting for the MZ gain slider was totaly wrong.
    Today I learned from the circuit, the full scale of the gain slider C1:PSL-MZ_GAIN gave us +/-10V at the DAC.
    This yield +/-1V to V_ctrl of the AD602 after the internal 1/10 attenuation stage.
    This +/-1V didn't correspond to -10dB~+30dB, but does -22dB~+42dB and is beyond the spec of the chip.

    The gain of AD602 is calculated by

G [dB] = 32 V_crtl + 10,  for -0.625 [V]< V_ctrl < +0.625 [V].

    In order to fix this I used the following commands which overrode the EPICS parameters.
    The tip of EGUF/EGUL is to know how much the gain (virtually) goes for the full scale of the DAC output. 

ezcawrite C1:PSL-MZ_GAIN.EGUF 42
ezcawrite C1:PSL-MZ_GAIN.EGUL -22
ezcawrite C1:PSL-MZ_GAIN.DRVH 30
ezcawrite C1:PSL-MZ_GAIN.DRVL -10
ezcawrite C1:PSL-MZ_GAIN.HOPR 30
ezcawrite C1:PSL-MZ_GAIN.LOPR -10

   and for the permanent change I modified the db file /cvs/cds/caltech/target/c1iool0/c1iooMZservo.db
   This will be active when cliool0 is rebooted.

# This yields the output limited to -6.25V ~ +6.25V, which corresponds to -10dB ~ +30dB
# modified by Koji Arai (29-Sept-2009)
grecord(ao,"C1:PSL-MZ_GAIN")
{
        field(DESC,"GAIN- overall pre-modecleaner servo loop gain")
        field(SCAN,"Passive")
        field(PINI,"YES")
        field(DISV,"1")
        field(DTYP,"VMIVME-4116")
        field(OUT,"#C3 S5 @")
        field(EGUF,"42")
        field(EGUL,"-22")
        field(PREC,"1")
        field(EGU,"dB")
        field(HOPR,"30")
        field(LOPR,"-10")
        field(DRVH,"30")
        field(DRVL,"-10")
        field(LINR,"LINEAR")
        field(OROC,"0")
        field(DOL,"0")
}

# previous code
grecord(ao,"C1:PSL-MZ_GAIN")
{
        field(DESC,"GAIN- overall pre-modecleaner servo loop gain")
        field(SCAN,"Passive")
        field(PINI,"YES")
        field(DISV,"1")
        field(DTYP,"VMIVME-4116")
        field(OUT,"#C3 S5 @")
        field(EGUF,"30")
        field(EGUL,"-10")
        field(PREC,"4")
        field(EGU,"Volts")
        field(HOPR,"30")
        field(LOPR,"-10")
        field(LINR,"LINEAR")
        field(OROC,"0")
        field(DOL,"0")
}