I measured the mode cleaner open loop gain.  It's around 60kHz with 29 degs of phase margin.

A 30-day trend of the PCDRIVE from the FSS.

 This problem has disappeared.  I don't know what it was. 

The first plot shows one of the symptoms.  The second plot is a similar section taken from a more normal acquisition sequence the night before.

All is not perfect, however, as now the handoff to RF CARM is not working.

LockAcq is back on track, with the full script working well.  Measurements in progress.

The problem has returned.  I still don't know what it is, but it's making me angry. 

I never actually figured out exactly what was wrong in entry 2162, but I managed to circumvent by changing the time sequence of events in the up script, moving the big gain increases in the common mode servo to the end of the script.  So the IFO can be locked again.

It won't start--it just sits at Waiting for EPICS BURT, even though the EPICS is running and BURTed.

[controls@c1omc c1omc]$ sudo ./omcfe.rtl
cpu clock 2388127
Initializing PCI Modules
3 PCI cards found
***************************************************************************
1 ADC cards found
        ADC 0 is a GSC_16AI64SSA module
                Channels = 64
                Firmware Rev = 3

***************************************************************************
1 DAC cards found
        DAC 0 is a GSC_16AO16 module
                Channels = 16
                Filters = None
                Output Type = Differential
                Firmware Rev = 1

***************************************************************************
0 DIO cards found
***************************************************************************
1 RFM cards found
        RFM 160 is a VMIC_5565 module with Node ID 130
***************************************************************************
Initializing space for daqLib buffers
Initializing Network
Waiting for EPICS BURT




From looking at the recorded data, it looks like the c1omc started going funny on the afternoon of Nov 5th, perhaps as a side-effect of the Megatron hijinks last week.

It works when megatron is shutdown.

Had been disconnected for about two weeks.  I found a partially seated 4-pin LEMO cable coming from the OSEM PD interface board. 

I've changed the watchdog rampdown script so it brings the SUS watchdogs to 220, instead of the 150 it previously targeted.  This is to make tripping less likely with the jackhammering going on next door.  I've also turned off all the oplev damping.

Locking has gone sour.  The CARM to MCL handoff, which is fairly early in the full procedure and usally robust, is failing reliably. 

As soon as the SUS-MC2_MCL gain is reduced, lock is broken.  There appears to be an instability around 10Hz.  Not sure if it's related.

 Five day minute trend.  FAST_F doesn't appear to have gone crazy.

 Whatever the locking problem was, the power of magical thinking has forced it to retreat for now.  The IFO is currently locked, having completed the full up script.  One more thing for which to be thankful.

Came in, found all front-ends down.

Keyed a bunch of crates, no luck:

Requesting coeff update at 0x40f220 w/size of 0x1e44
No response from EPICS 

Powered off/restarted c1dcuepics.  Still no luck.

Powered off megatron.  Success!  Ok, maybe it wasn't megatron.  I also did c1susvme1 and c1susvme2 at this time.

BURT restored to Nov 26, 8:00am

But everything is still red on the C0_DAQ_RFMNETWORK.adl screen, even though the front-ends are running and synced with the LSC.  I think this means the framebuilder or the DAQ controller is the one in trouble--I keyed the crates with DAQCTRL and DAQAWG a couple of times, with no luck, so it's probably fb40m.    I'm leaving it this way--we can deal with it tomorrow.

 A "Data Receiving Error" usually indicates a problem with the framebuilder/testpoint manager, rather than the front-end in question.  I'd bet there's a DTT somewhere that's gone rogue.

Taking  a cue from entry 2346, I immediately went for the nuclear option and powered off fb40m.  Someone will probably need to restart the backup script.

There's a large broadband increase in the MC_F spectrum.  I'm not totally sure it's real--it could be some weird bit-swapping thing.  I've tried soft reboots of c1susvme2 and c1iovme, which haven't helped.  In any case, it seems like this is preventing any locking success today.  Last night it was fine.

 Rebooting c1iovme (by keying off the crate, waiting 30 seconds, and then keying it back on and restarting) has resolved this.  The frequency noise is back to the 'usual' trace.

Koji, Jenne, Rob

We found that the RCPID servo "setpoint" was not in the relevant saverestore.req file, and so when c1psl got rebooted earlier this week, this setting was left at zero.  Thus, the RC got a bit chilly over the last few days.  This channel has been added. 

Also, RCPID channels have been added (manually) to conlog_channels. 

This is a (sort of) known problem with the EPICS computers: it's generally called the 'sticky slider' problem, but of course it applies to buttons as well.  It happens after a reboot, when the MEDM control/readback values don't match the actual applied voltages.  The solution (so far) is just to `twiddle' the problematic sliders/button.  There's a script somewhere called slider_twiddle that does this, but I don't remember if it has PSL stuff in it.  A better solution is probably to have an individual slider twiddle script for each target machine, and add the running of that script to the reboot ritual in the wiki.

 oops, my bad.  I cranked the 33MHz modulation depth and forgot to put it back.  The slider should go back to around 3. 

 I suggest "ITF" for the model name.

 Has anyone tried pushing the "reset" button on the Uniblitz driver?

Lock acquisition is proceeding smoothly for the most part, but there is a very consistent failure point near the end of the cm_step script.

Near the end of the procedure, while in RF common mode, the sensing for the MCL path of the common mode servo is transitioned from a REFL 166I signal which comes into the LSC whitening board from the demodulator, to another copy of the signal which has passed through the common mode board, and is coming out of the Length output of the common mode board.  We do this because the signal which comes through the CM board sees the switchable low-frequency boost filter, and so both paths of the CM servo (AO and MCL) can get that filter switched on at the same time.

The problem is occurring after this transition, which works reliably.  However, when the script tries to remove the final CARM offset, and bring the offset to zero, lock is abruptly lost.  DARM, CM, and the crossover all look stable, and no excess noise appears while looking at the DARM, CARM, MCF spectra.  But lock is always lost right about the same offset. 

Saturation somewhere?

I used the XARM as a reference to measure the frequency noise after the MC.  It's huge around 4kHz--hundreds of times larger than the frequency noise the MC servo is actually squashing.  This presents a real problem for our noise performance.

An elog search reveals that this noise has been present (although not calibrated till now) for years.  We're not sure what's causing it, but suspicion falls on the piezojena input PZTs. 

I didn't bother too much about it before because we previously had enough common mode servo oomph to squash it below other DARM noises, and I didn't worry too much about stuff at 4kHz..  Now that we have a weaker FSS and thus much weaker CM servo, we can't squash it, and the most interesting feature of our IFO is at 4kHz. 

I'll measure the actual voltage noise going to the PZTs.  I remember doing this before and concluding it was ok, but can't find an elog entry.  So this time maybe I'll  do it right.

This level of frequency noise has not changed, but we now have increased common mode servo gain and so it's not as huge of a deal, although we should still probably do something about it. 

Attached is a plot of the piezojena noise measurement, estimated into Hz, along with another measurement of frequency noise as described above. 

To get the piezojena voltage noise into Hz, I estimated the PZTs within have a flat 2 micron/V response (based on a rough knowledge of their geometry and assuming a 10 milliradian / 150V steering range).  This is the voltage noise with the PZTs operating in closed loop mode, which is how we normally run them.  This plot also ignores the transfer function of the Pomona box, as we are mainly looking at noise in the kHz band.  I think this plot shows that these PZTs are a good candidate for creating this frequency noise, especially near their mechanical resonances (the manual says the unloaded resonances are in the 3-4kHz range).   

I've been operating one DOF of the piezojenas in open loop mode for a couple of weeks now, and the feared drift has not been a problem at all.  If we plan to keep using these after the upgrade, we should definitely put some big resistors in series at the outputs and operate them in open loop mode.

Also attached is a plot of RF DARM noise, with a frequency noise spectrum.  That spectrum is a REFL 2I spectrum put into DARM units using a measured TF (driving MC_L and measuring REFL 2I and DARM_ERR), and then put into meters using the same DARM calibration as used for the DARM curve.

We can't read minute trends from either Dataviewer or loadLIGOData from before 11am this morning. 

fb:/frames>du -skh minute-trend-frames/
 106G   minute-trend-frames


So the frames are still on the disk.  We just can't get them with our usual tools (NDS).

 Trying to read 60 days of minute trends from C1:PSL-PMC_TRANSPD yields:

Connecting to NDS Server fb40m (TCP port 8088)
Connecting.... done
258.0 minutes of trend displayed
read(); errno=9
read(); errno=9
T0=09-06-06-22-34-02; Length=5184000 (s)
No data output.


Trying to read 3 seconds of full data works.

Second trends are readable after about 4am UTC this morning, which is about 9 pm last night.

Recently the watch script was having difficulty grabbing a lock for more than a few seconds.  Rob discovered that the violin notch filters which were activated in the script were causing the instability.  We're not sure why yet.  The script seems significantly more stable with that step commented out.

we did a hard reboot of c1susvme1, c1susvme2, c1sosvme, and c1susaux.  We are hoping this will fix some of the weird suspension issues we've been having (MC3 side coil, ITMX alignment).

We were stymied early in the evening by a surreptitiously placed, verbo-visually obfuscated command in the drstep script. 

Restarting the framebuilder didn't work, but the problem now appears to be fixed.

Upon reflection, we also decided to try killing all open DTT and Dataviewer windows.  This also involved liberal use of ps -ef to seek out and destroy all diag's, dc3's, framer4's, etc.

That may have worked, but it happened simultaneously to killing the tpman process on fb40m, so we can't be sure which is the actual solution.

To restart the testpoint manager:

what you type is in bold:

rosalba> ssh fb40m

fb40m~> pkill tpman

The tpman is actually immortal, like Voldemort or the Kurgan or the Cylons in the new BG.  Truly slaying it requires special magic, so the pkill tpman command has the effect of restarting it.

In the future, we should make it a matter of policy to close DTTs and Dataviewers when we're done using them, and killing any unattended ones that we encounter.

We found that someone had set the name of megatron to scipe11. This is the same name as the existing c1aux in the op440m /etc/hosts file.

We did a /sbin/shutdown on megatron and the OMC now boots.

Please: check to see that things are working right after playing with megatron or else this will sabotage the DR locking and diagnostics.

The MOPA is taking the long weekend off.

Steve went out to wipe off the condensation inside the MOPA and found beads of water inside the NPRO box, perilously close to the PCB board.  He then measured the water temperature at the chiller head, which is 6C.  We decided to "reboot" the MOPA/chiller combo, on the off chance that would get things synced up.  Upon turning off the MOPA, the neslab chiller display immediately started displaying the correct temperature--about 6C.  The 22C number must come from the MOPA controller.  We thus tentatively narrowed down the possible space of problems to: broken MOPA controller and/or clog in the cooling line going to the power amplifier.  We decided to leave the MOPA off for the weekend, and start plumbing on Tuesday.  It is of course possible that the controller is the problem, but we think leaving the laser off over the weekend is the best course of action.

 To fix open-mx connection to c1ioo, had to restart the mx mapper on fb machine. Command is /opt/mx/sbin/mx_start_mapper, to be run as root. Once this was done, omx_info on c1ioo computer showed fb:0 in the table and mx_stream started back up on its own. 

The PSL temperature box has returned to service, with some circuit modifications. The 1k resistors on all the temp. sensor inputs (R3, R4, R7, R8, R12, R12) were changed to 0 Ohm. Also, the 10k resistors R26, R28, R29, and R30 were changed to 10.2k metal film. The DCC document will be updated shortly. There is now an offset in the MINCOMEAS channel compared to the others, which will be corrected in the morning after looking at the overnight trend.

 Forgot to do this in May. Have just changed the values in the psl.db file now as well as updating them live via Probe.

To make the appropriate change, I took the measured offset (5.31 deg) and added 2x this to the EGUF and EGUL field for the MINCO_MEAS channel. (see instructions here)

Committed the .db file to the SVN.

attached plot shows 8 days of trend with 5.31 degC added to the black trace using the XMGRACE Data Set Transformations

 Xinerama support has been enabled on rossa using nvidia-settings.

added name server 192.169.113.20 as the first entry in /etc/resolv.conf

changed the host IPs in /etc/hosts to 192.168.xxx.yyy

made:

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhos6

as the first two lines of /etc/hosts

/cvs/cds mounts

on ethernet, DNS look-up works without the explicit host definitions in /etc/hosts,

but those entries are needed for wifi only connection.

we set the offsets for the MCWFS DC and for the MCWFS demod outputs and then turned off the lights put the MZ at half fringe and then centered the spots on the MCWFS heads.

The MCREFL beam looks symmetric again and the MC REFL power is low. 

Kiwamu and I setup a serial port terminal for receiving data from TC200 via a RS-232 USB interface. It was done using a Python code. Some command definitions need to be done to get the output from TC-200.

We (Rana and I) are re-assembling the temperature controls on the seismometer to attempt PID control and then improve it using reinforcement learning.

We tried to re-assemble the connections for the heater and in-loop temperature sensor on the can that covers the seismometer.

We fixed (soldered) two of the connections from the heater circuit to the heater, but did not manage to get the PID working as one of the wires attached to the MOSFET had come off. Re-soldering the wire would be attempted tomorrow.

Equipment for undertaking all this is still left at the X-end of the interferometer and will be cleared soon.

Two out of the four over-head fluorescent lights in the X end of the interferometer were flickering today.

After removing all the clamping screws from the heater circuit board, I soldered the wire connecting IRF630 to the output of OP27, which had come off earlier. This can only be a temporary fix as the wire was not long enough to be able to make a proper solder joint. I also tried fixing two other connections which were also almost breaking.

After re-assembling everything I found out that one of the LEDs was not working. The most likely cause seems to be an issue with LM791, LM 781 or the LED itself. Due to the positioning of the wires, I was unable to test them today but will try again possibly tomorrow.

Equipment used for this is still lying at the X end.

Earlier today I cleared up most of the equipment at the X end near the seismometer to make the area walkable. 

In the process, I removed the connections to the temperature sensor and placed the wires on top of the can.

[Shruti, Rana]

- At the X end, we set up the network analyzer to begin measurement of the AM transfer function by actuation of the laser PZT.

- The lid of the PDH optics setup was removed to make some checks and then replaced.

- From the PDH servo electronics setup the 'GREEN_REFL' and 'TO AUX-X LASER PZT' cables were removed for the measurement and then re-attached after.

- The signal today was too low to make a real measurement of the AM transfer function, but the GPIB scripts and interfacing was tested.