The previous "+15" was Vctrl = 0.25 [V]. Which was +18 dB.

Probably there is the same mistake for the PMC gain slider. Possibly on the FSS slider, too???

We had a redo of elog entry 975 tonight.  The noisy OSEM was fixed by jiggling the rack end of the long cable.  Don't know exactly where--I also poked around the OSEM PD interface board.

In the attached PDF the reference trace is the noisy one.

much grief.  somehow a burt restore of c1iscepics failed to work, and so the LSC XYCOM settings were not correct.  This meant that the LSC whitening filter states were not being correctly set and reported, making it difficult to lock for at least the last week or so.

It's been a miserable week for lock acquisition, with each night worst than the last.  The nadir was around Sunday night, when I couldn't even get a PRM to lock stably, which meant that the auto-alignment scripts could not finish successfully.  It now appears that was due to some XYCOM mis-settings. 

We've also been having problems with timing for c1susvme2.  Attached is a one-hour plot of timing data for this cpu, known as SRM.  Each spike is an instance of lateness, and a potential cause of lock loss.  This has been going on for a quite a while.

Tonight we also encountered a large peak in the frequency noise around 485 Hz.  Changing the MZ lock point (the spot in the PZT range) solved this.

I'm blaming Zach on this one, for trying to upload a pic into the ATF elog (even though he claims it was small....)  Blame assigned: check.  Elog restarted: check.

[Sanjit, Jenne]

The up and down scripts accessible from the OAF (still C1:ASS-TOP) screen are now totally functional and awesome.  They are under the blue ! button.  The up script can either be for the Seismometers, or the Accelerometers at this time.  The only difference between these 2 is which burt restore file they look at:  the seismometer version puts all 7 seismometer channels in the PEM selecting matrix, while the accelerometer version puts the 6 accelerometer channels in that matrix.  Both scripts also turn on HP_1mHz filters in the ERR_EMPH filter bank and all of the witness filter banks, and the AA32 and AI32 filters in ERR_EMPH, CORR and PEM filter banks.  This makes all of the starting filters the same between the witness paths and the error path.

If you want to use a different combination of sensors, run one of the up scripts, then change the PEM matrix by hand. 

The down script disables the output to the optics, and resets the adapted filter coefficients.  DO NOT use this script if you're trying to "pause" the filter to take some nice long averages.

Good progress in IFO locking tonight, with the arm powers reaching about half the full resonant maximum. 

Still to do is check out some weirdness with the OMC DAC, fix the wireless network, and look at c1susvme2 timing.

Earthquake  of magnitude 5.0  shakes ETMY loose.

MC2 lost it's damping later.

MZ stayed unlocked. Now It was relocked.

 The EQ did not change the input beam pointing. All back to normal, except MC2 wachdogs tripped again.

 Round 3 for the day of MC2 watchdogs tripping.

I will investigate the MZ board. I will unlock MZ (and MC).

 I've watchdogged all the suspensions while I mess around with computers.  If no one else is using the IFO, we can leave them undamped for a couple of hours to check the resonant frequencies, as long as I don't interrupt data streams with my computer hatcheting.

Attached is a 3 day trend of SRM CPU timing info.  It clearly gets better (though still problematic) at some point, but I don't know why as it doesn't correspond with any work done.  I've labeled a reboot, which was done to try to clear out the timing issues.  It can also be seen that it gets worse during locking work, but maybe that's a coincidence.

MZ work has been done. I did not change anything on the circuit.

Recently we observed that the MZ PZT output was sticking at a certain voltage. I found the reason.
Shortly to say "we must return the PZT Ramp offset to 0, after the lock"

I am going to write a MZ auto lock script someday, to do it automatically.

According to the resister values used in the circuit, the PZT HV output voltage is determined by the following formula:

V_PZT = 150 - 12 V_ctrl - 24 Vramp

Here the ramp voltage Vramp moves from -10V to +10V, the feedback control voltage V_ctrl moves from -13V to +13V.
The baseline offset of 150V is provided in the circuit board.

When V_ramp is 0, V_PZT runs from 0 to 300. This is just enough for the full scale of the actual V_PZT range,
that is 0V~280V.

If any Vramp offset is given, V_PZT rails at either side of the edges. This limits the actual range of the PZT out.

This is not nice, but is what happened recently.

Ops. I restored the damping of the suspensions at around 16:30.

More progress with locking tonight, with initial acquisition and power ramps working.  The final handoff to RF CARM still needs work.

I found the wireless router was unplugged from the network--just plugging in the cable solved the problem.  For some reason that RJ45 connector doesn't actually latch, so the cable is prone to slipping out of the jack.

The attached shows the 200 day '10-minute' trend of the CPU meters and also the room temperature.

To my eye there is no correlation between the signals. Its clear that c1susvme2 (SRM LOAD) is going up and no evidence that its temperature.

It got worse again, starting with locking last night, but it has not recovered.  Attached is a 3-day trend of SRM cpu load showing the good spell.

Here's the gist of the requirements on the 5x frequency multiplier for the upgrade (see attachemnt - despite the preview down here in the elog, they're 3 pages).

An extended version is available on the wiki.

A more complete document is under work and will soon be available on the wiki as well.

I took the output of the OMC DAC and plugged it directly into an OMC ADC channel to see if I could isolate the OMC DAC weirdness I'd been seeing.  It looks like it may have something to do with DTT specifically.

Attachment 1 is a DTT transfer function of a BNC cable and some connectors (plus of course the AI and AA filters in the OMC system).  It looks like this on both linux and solaris.

Attachment 2 is a transfer function using sweepTDS (in mDV), which uses TDS tools as the driver for interfacing with testpoints and DAQ channels. 

Attachment 3 is a triggered time series, taken with DTT, of the same channels as used in the transfer functions, during a transfer function.  I think this shows that the problem lies not with awg or tpman, but with how DTT is computing transfer functions. 

I've tried soft reboots of the c1omc, which didn't work.   Since the TDS version appears to work, I suspect the problem may actually be with DTT.

Late last night after the ETMY settled down from the quake I made some more progress in locking, with the handoff to RF CARM succeeding once.  The final reduction of the CARM offset to zero didn't work, however.

Working well tonight: the handoff of CARM to RF (REFL2I), successful reduction of CARM offset to zero, and transition control of MCL path to the OUT1 from the common mode board.  All that's left in lock acquisition is to try and get the common mode bandwidth up and the boost on.

The PSL output looks smaller than the incident. Try to FSS Slow actuator adj of -5.6 (nominal), instead of -3.5.

I have been commissioned to take pictures of the PSL table so that it can be diagrammed. I am starting now (1:42 pm, 10/5/09).

All done (for now). That wasn't so bad, was it?

In the revival of the experiement length measurement for the recycling cavities, I turned the auxiliary NPRO back on. The shutter is closed.

I also recollected all the equipment of the experiment after that during the summer it had been scattered around the lab to be used for other purposes (Joe and Zach's cameras and Stephanie and Koji's work with the new EOM).

[Jenne, Sanjit]

As per Matt's instructions in his OAF document (elog 395) in the Tuning section, Sanjit and I took a transfer function measurement from the output of the OAF system, to the input.  i.e. we're trying to measure what happens out in the real world when we push on MC1, and how that is fed back to the input of our filter as MC_L.  The game plan is to measure this transfer function, and read off the phase at the nyquist frequency, and use this value to calculate the appropriate sample-and-hold delay to be used in the OAF.  The downsample rate for the OAF is 32, so that we're running at 64Hz instead of the 2048Hz of the front-end.  Thus, our Nyquist frequency is 32Hz.

                            DownSampleRate

Phase@Nyquist * ------------------------

                                    180

In the attached figure we do a swept sine from CORR_EXC to ERR_EMPH_OUT to determine the transfer function.  Here, we turn off all of the filters in both the CORR and EXC banks, because those are already matched/taken into account in the PEM filter banks. 

Using the cursor on DTT, we find that the phase at 29.85Hz is -228.8deg, and at 37.06Hz is -246.0deg.  Extrapolating, this means that at 32Hz, we expect about -234deg phase.  Using our handy-dandy formula, this means that we should try a delay of 41 or 42 (41.6 is between these two...) 

We'll give this a shot!

I set the FSS slow actuator adj to -5.6 at the lunch time. It gave a little help at that time. Now max of the MC Trans is comming back somehow. I hope the MC Trans level is as good as before, if the HEPA is slowed down.

Lock acquisition working well tonight.  Was able to engage CM boost (not superboost) with bandwidth of ~10kHz.  Also succeeded once in handing off DARM to DC readout.

I'm back to terrorize the PSL table again. The pictures I took yesterday were rubbish--today I'm using a clamp that Steve was nice enough to loan me. I'm starting now, at 10:09 am.

 As Rana pointed out to me last night, I was using continuous phase, which is not good.  When using regular phase, I find: (29.85Hz, 131.216deg), (37.06Hz, 113.963deg), so extrapolating gives (32Hz, 126.07deg).  Plugging this in to our handy-dandy formula, we get a delay of 22.4, so we should try both 22 and 23.

 Here's a plot of the spectra of the seismometers and MCL. The coherence shows which axes are aligned right now: MC1_X is coherent with GUR_NS which means that its mis-oriented.

I've now swapped the "MC1" cables: so the old "NS" now goes into EW and the old EW now goes into NS. VERT is unchanged.

Also fixed the channel names - the Guralp previously named MC1 is now GUR1 and the other one is GUR2. Also no more EW, NS, & VERT. Its all XYZ.

DAQD restarted with the new channel names.

I remeasured the OAF time delay using the OAF-TF template from the Templates/ directory.

Troublingly, I found the MC1 dewhitening switches set OFF - please make sure that the MC1 dewhitening is back ON after each OAF tuning so that the interferometer locking is not hosed.

The OAF-TF template had the excitation amplitude set ~20x too high. I reduced it and the coherence was still > 0.95. The phase at 32 Hz was still ~126 deg as Jenne had measured, but since the phase at DC is 180 deg, the overall phase lag is just 180-126 = 54 deg. So the delay should be 54/180 * 32 = 9.7 => 10. Luckily, Jenne is working on an instructional manual for OAF that will make all of this crystal clear.

 I spiffed up the order of the cables / sensors plugged into the PEM ADCU.  Now all of the seismometers are labeled as Rana left them, and the 2 Guralp's have their sets of 3 channels next to eachother in channel-number-land.  None of the accelerometer names/cabling have changed recently.  In the table, Cable-label refers to the physical tag tied to the end of the cables plugged into the ADCU...they are meant to be descriptive of what seismometer channels they are hooked up to, and then the names change to something useful for us when they come into the DAQ system.  Also, the labels of input channels on the ASS_TOP_PEM screen have been updated accordingly.

[Rana, Jenne]

There is some craziness going on with the delay in the PEM path for the OAF.  We plot the difference between the C1:PEM-SEIS_GUR1_X and C1:ASS-TOP_PEM_10.  These are physically the same channel, plugged into the PEM ADCU, and then the signal is used as a regular PEM channel, and is also sent to the ASS computer and used there for the OAF system.  As you can see in the blue trace on the bottom plot, there is a huge amount of delay, and it's very noisy.  We also plot the _GUR2_X / ASS-TOP_PEM_2 pair (red), and it has a similar amount of delay, but it is not nearly as fuzzy and noisy.  For comparison, we plot the SUS-MC2_MCL (which is identical to IOO-MC_L) and ASS-TOP_ERR_MCL pair (green), and they don't have any big overall delay problems, so it's not totally a problem with the signals getting to the ASS computer.

This problem was present during/after all of the following attempts to fix it:

* The sample rate on the ASS computer is 2048.  The PEM channels were being acquired the ADCU at 512.  We changed the ADCU sampling rate to 2048 to match.

* We soft rebooted the ASS computer, in case it was a timing problem.

* Doing a "sudo shutdown -r now" while logged in as controls.

We might also try resetting/power cycling c0dcu in the morning.  Alex has been emailed to help us try to figure this out.

In other news, the time delay that we measure from the plot gives us 180degrees in ~210Hz.  This corresponds to a little more than 2msec of delay, with the C1:ASS version lagging behind the C1:PEM version.  (2 samples at 840Hz) Converting to the 2048 sampling rate, we have a delay of 4.8, so 5 front-end cycles.  Since Rana measured this morning that the delay indicated by the transfer function is 10 cycles, and this delay shows that the ASS lags the actual seismometer signal by 5 cycles, we should subtract this 5 from the 10 from the transfer function, giving us a final sample-and-hold delay of 5.  Coincidentally(?), 5 is the delay that was found in the C1:ASS-TOP screen, after it's one year of dormancy.  The point of the delay feature in the code is to help match the delay in the two signal paths: the PEM path and the output path of the filter.  Since the output has a lag of 10, and the PEM path has a lag of 5, to make them match, we artificially put in a delay of 5.

At around 9:45 the RFM/FB network alarm went off, and I found c1asc, c1lsc, and c1iovme not responding. 

I went out to hard restart them, and also c1susvme1 and c1susvme2 after Jenne suggested that.

c1lsc seemed to have a promising come back initially, but not really.  I was able to ssh in and run the start command.  The green light under c1asc on the RFMNETWORK status page lit, but the reset and CPU usage information is still white, as if its not connected.   If I try to load an LSC channel, say like PD5_DC monitor, as a testpoint in DTT it works fine, but the 16 Hz monitor version for EPICs is dead.  The fact that we were able to ssh into it means the network is working at least somewhat.

I had to reboot c1asc multiple times (3 times total), waiting a full minute on the last power cycle, before being able to telnet in.  Once I was able to get in, I restarted the startup.cmd, which did set the DAQ-STATUS to green for c1asc, but its having the same lack of communication as c1lsc with EPICs.

c1iovme was rebooted, was able to telnet in, and started the startup.cmd.  The status light went green, but still no epics updates.

The crate containing c1susvme1 and c1susvme2 was power cycled.  We were able to ssh into c1susvme1 and restart it, and it came back fully.  Status light, cpu load and channels working.  However I c1susvme2 was still having problems, so I power cycled the crate again.  This time c1susvme2 came back, status light lit green, and its channels started updating.

At this point, lacking any better ideas, I'm going to do a full reboot, cycling c1dcuepics and proceeding through the restart procedures.

Power cycling c1dcuepics seems to have fixed the EPICs channel problems, and c1lsc, c1asc, and c1iovme are talking again.

I burt restored c1iscepics and c1Iosepics from the snapshot at 6 am this morning.

However, c1susvme1 never came back after the last power cycle of its crate that it shared with c1susvme2.  I connected a monitor and keyboard per the reboot instructions.  I hit ctrl-x, and it proceeded to boot, however, it displays that there's a media error, PXE-E61, suggests testing the cable, and only offers an option to reboot.  From a cursory inspection of the front, the cables seem to look okay.  Also, this machine had eventually come back after the first power cycle and I'm pretty sure no cables were moved in between.

 I had a go at trying to bring c1susvme1 back online.  The first few times I hit the physical reset button, I saw the same error that Joe mentioned, about needing to check some cables.  I tried one round of rebooting c1sosvme, c1susvme2 and c1susvme1, with no success.  After a few iterations of jiggle cables/reset button/ctrl-x on c1susvme1, it came back.  I ran the startup.cmd script, and re-enabled the suspensions, and Mode Cleaner is now locked.  So, all systems are back online, and I'm crossing my fingers and toes that they stay that way, at least for a little while.

The c1ass computer, which is now used for the OAF system, has many remnants from the days when it was actually used as an ASS.  These PIT and YAW filter banks and other things were taking up a lot of unnecessary space, so I deleted them in the ass.mdl file.  These files are all backed up, so we can always revert back to an older version when we want some Alignment Stabilization again someday.  I then did a make ass, following the instructions on the 40m Wiki -> Computers and Scripts -> Simulink to Front-End Code page.  Rana moved some things around, most notably all of the things (like the ASS screens) which were only in ...../users/alex/.... are now in ....../caltech/cds/advLigo/..... .  This required a few restarts of the c1ass machine (after a couple different versions of the simulink diagram....one to make sure we knew how to do it, and then again actually deleting the unused portions).

The big lesson of the night was that there are 2 signal paths for the PEM channels.  As is shown in Figure 3 in the mevans document, the PEM channels get the matching filters when they go to the adaptation algorithm, but when they go to the FIR filter, they do not get the matching filters. This is implemented by taking the output of the giant PEM matrix, and having a duplicate of each of the channels "selected for adaptation", one which gets filtered through the PEM_N_ADPT banks, and one which goes straight (in code-land) to the FIR filter.  So, it seems like all the filters which we had been including in the input side of the matrix for matching purposes need to be put in the output side.  One of the AA32 filters needs to stay in the input side, for actual anti imaging of the PEM channels, then we put the AA32 and AI32 which are for matching the ERR_EMPH and CORR filter banks up in the PEM_N_ADAPT banks.  Rana and I made these filters, and they are now turned on appropriately with the OAF down script (so that all the filters are ready and waiting for the OAF to be turned on).

A little success with getting the 3Hz peak reduced, but not a lot beyond that.  Tomorrow I'll put the accelerometers back where they used to be to see if they help out at all.

I used the XARM as a reference to measure the frequency noise after the MC.  It's huge around 4kHz--hundreds of times larger than the frequency noise the MC servo is actually squashing.  This presents a real problem for our noise performance.

An elog search reveals that this noise has been present (although not calibrated till now) for years.  We're not sure what's causing it, but suspicion falls on the piezojena input PZTs. 

I didn't bother too much about it before because we previously had enough common mode servo oomph to squash it below other DARM noises, and I didn't worry too much about stuff at 4kHz..  Now that we have a weaker FSS and thus much weaker CM servo, we can't squash it, and the most interesting feature of our IFO is at 4kHz. 

I'll measure the actual voltage noise going to the PZTs.  I remember doing this before and concluding it was ok, but can't find an elog entry.  So this time maybe I'll  do it right.

Last night while noise hunting, Rana found that the MC2 trans beam has been left for an unknown length of time just hitting the side of the black enclosure box.  Today I put a brand-spankin'-new razor dump on the MC2 table, to dump the beam.

[Sanjit, Jenne]

The set of 6 accelerometers which were semi-randomly placed underneath the MC2 tank are now back into 2 separate sets of 3 - one set at MC2 and one set at MC1.  The channel names once again reflect reality, i.e. MC1_Y is actually under the MC1 tank, and aligned with the y direction.  Also, the Guralp under MC1 was moved a little bit to the left, because Sanjit wanted to put the accelerometers where the seismometer had been. 

Some of these channels are not like the others.

 Last week, Alex recompiled the c1susvme2 code without the decimation filters for the OUT16 channels, so these channels are now as aliased as the rest of them.  This appears to have helped with the timing issues: although it's not completely cured it is much better.  Attached is a five day trend.

Last night, 2+ hour lock, probably broken by me driving too hard (DARM_EXC).

--Apologies for the late post--

I was at the PSL table taking an inventory of the components for a while after Koji, Steve, and Kiwamu were there. I set the HEPAs back to 20% when I left (assuming that they were turned up when the compartment was opened).

I am at the PSL table taking what is hopefully the last set of pictures for the diagram. Woohoo.

 I'm out, HEPAs are back at 20%.