40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 41 of 341  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  4823   Wed Jun 15 12:16:37 2011 JamieUpdateLockingMICH noise budget?

Quote
If you tilt your head sideways, you will notice that in this plot (totally uncalibrated, as yet), the BLACK trace, which is my white-light measurement of the AS55 shot noise is above the AS55Q noise when the Michelson is locked (true only at low frequency).  You will also notice that the same appears to be true for the Whitening Filter + Antialiasing Filter + ADC noise (GRAY trace).  Since Black, Gray, Pink and Green should all have the same calibration factor (a constant), calibrating the plot will not change this.  Brown and Blue are the MICH_OUT (aka MICH_CTRL) for dark and bright fringes, respectively.

Hey, Jenne.  I think there are a couple of things.  First, you're missing a PD dark noise measurement, which would be useful to see.

But I think the main issue is that it sounds like all of your closed loop measurements are done with the in-loop PD.  This means that everything will be suppressed by the loop gain, which will make things look like they have a noise lower than the actual noise floor.

  4833   Fri Jun 17 17:02:15 2011 JamieUpdateSUSETMX/ETMY binary output modules (re)installed, not yet tested

I have installed a new binary output module in ETMY, where there was none previously.  It is installed, powered (with working LEDs), hooked up (to the binary output card and the cross connect), but it hasn't been fully tested yet.

I also re-installed the binary output module in ETMX, with newly modified power-indicator LEDs.

Both modules are fully installed, but they have not yet been fully tested to confirm that they are indeed switching the whitening and de-whitening filters.

  4837   Mon Jun 20 09:28:19 2011 JamieUpdateCDSShutting down low-voltage DC power in 1X1/1X2 racks

In order to install the BO module in 1X2, I need to shut down all DC power to the 1X1 and 1X2 racks.

  4838   Mon Jun 20 10:45:43 2011 JamieUpdateCDSPower restored to 1X1/1X2 racks. IOO binary output module installed.

All power has been restored to the 1X1 and 1X2 racks.  The modecleaner is locked again.

I have also hooked up the binary output module in 1X2, which was never actually powered.  This controls the whitening filters for MC WFS.  Still needs to be tested.

  4859   Wed Jun 22 18:50:45 2011 JamieSummaryGeneralJuly 2011 vent plan

Kiwamu and I have started to put together a vent plan on the 40m wiki:

http://blue.ligo-wa.caltech.edu:8000/40m/vent/201107

We will keep working on this (there's still a *lot* to fill in), but please help fill in the plan by adding questions, answers, procedures, preparations, etc.

 

  4869   Thu Jun 23 22:00:22 2011 JamieUpdateSUSburt snapshot

I recorded a burt snapshot of these settings: /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2011/Jun/23/21:40

  4902   Tue Jun 28 21:05:05 2011 JamieUpdateSUSSUS control model updated

I have updated the sus_single_control model, adjusting/cleaning up/simplifying the LSC/POS input signals, and routing new signals to the lockins. Notably one of POS inputs to the part ("lockin_in") was eliminated (see below).

The 6 inputs to the TO_COIL output matrix are now:

LSCPOS + OFFSET + ALT_POS_IN
ASCPIT + OFFSET + SUSPIT + OLPIT
ASCYAW +OFFSET + SUSYAW + OLYAW
SIDE
LOCKIN1
LOCKIN2

The ALT_POS input is used only by the ETMs for the green locking. Just outside of the sus_single_control library part in the ETM models are the green locking controls, consisting of the ETM?_ALS filter bank and the ETM?_GLOCKIN lockin, the outputs from which are summed and fed into the aforementioned ALT_POS input.

As for the SUS lockins (LOCKIN1 and LOCKIN2 in the library model), their input matrix now gets the direct inputs from the OSEMS (before filtering) and the outputs to the coils, after all filtering. These will aid in doing binary output switching tests.

All suspension models (c1sus, c1scx, c1scy) have been rebuild and restarted so that they reflect these changes.

  4903   Tue Jun 28 22:07:46 2011 JamieHowToSAFETYEurocrate extender card fried (ie. Jamie did a very bad thing)

IMPORTANT ELECTRONICS SAFETY LESSON TO FOLLOW

Yesterday, I fried the +15 V power supply rail on one of the Eurocrate extender cards while I was checking out the binary switching in the 1X5 rack.  I will describe what I did it in the hopes that everyone else will be less stupid than me.

I wanted to monitor the voltage across a resistor on the suspension OSEM whitening board.  Since I knew that both sides of the resistor would be at non-zero voltage (including possibly at the power-supply rail), I used a battery-operated scope with floating inputs, so that the scope would not try to pull the probe shield to ground.  That should be OK, although not recommended, as you'll see, because you must be very careful to make sure that the scopes inputs are indeed floating.

Let's call the original signal 'A'.  The trouble came when I then connected another signal (B), whose shield was connected to the ground on the whitening board, to the scope.  Apparently the grounds on the scope inputs are connected, or were in the configuration I was using.  When I connected the signal B, B's ground shorted A's shield to ground, which had been sitting at the +15V rail.  That short circuit then fried the +15V supply line on the extender card I was using (escaping magic smoke was detected).  Thankfully this only blew the extender card, and not the Eurocrate or the Eurocrate power supply or the whitening board or the scope etc, all of which would have been much worse.

The moral of the story is to be very careful when connecting power supply voltages to the shield or ground of a scope.  In short, don't do it.  I didn't ultimately need to, since I could have found other ways to measure the same signal.

 

  4904   Tue Jun 28 22:36:04 2011 JamieUpdateSUSChecking binary switching of SUS whitening filter

I have been checking the binary output switching for the SUS whitening filters. It appears that the whitening switching is working for (almost) all the vertex suspensions (BS, ITMX, ITMY, PRM, SRM), but not for the ETMs.

The table below lists the output from my switch-checking script (attached). The script uses the SUS digital lockin to drive one coil and measure the same coil's OSEM response, repeating for each coil/OSEM pair. I used a lockin drive frequency of about 10 Hz, at which the whitening filter should have 10 db of gain.

All but one of the vertex OSEMS show the proper response (~10db gain at 10Hz) when the whitening is switched on from the digital controls. ITMY UL appears to not be switching, which I fear is due to my electronics fail noted in my previous log post.  The ETMs are clearly not switching at all.

I will try to get the ETM switching working tomorrow, as well as try to asses what can be done about the ITMY UL switch.  After that I will work on confirming the coil drive dewhite switching.

lockin settings

freq: 10.123 Hz
amp: 10000
I/Q filters: 0.1 Hz LP, 4-pole butterworth

response

BS
ul : 3.31084503062 = 10.3987770676 db
ll : 3.34162124753 = 10.4791444741 db
sd : 3.43226254574 = 10.7116100229 db
lr : 3.28602651913 = 10.3334212798 db
ur : 3.29361593249 = 10.3534590969 db

ITMX
ul : 3.37499773336 = 10.5654697099 db
ll : 3.2760924572  = 10.3071229966 db
sd : 3.13374799272 =  9.9212813757 db
lr : 3.28133776018 = 10.3210187243 db
ur : 3.37250879937 = 10.5590618297 db

ITMY
ul : 0.99486434364 = -0.0447226830807 db
ll : 3.39420873724 = 10.6147709414 db
sd : 3.88698713176 = 11.7922620572 db
lr : 3.357123865   = 10.5193473069 db
ur : 3.37876008179 = 10.5751470918 db

PRM
ul : 3.26758918055 = 10.2845489876 db
ll : 3.32023820566 = 10.4233848529 db
sd : 3.25205538857 = 10.2431586766 db
lr : 3.24610681962 = 10.227256141  db
ur : 3.31311970305 = 10.4047425446 db

SRM
ul : 3.30506423619 = 10.3835980943 db
ll : 3.28152094133 = 10.3215036019 db
sd : 3.08566647696 =  9.7869796462 db
lr : 3.30298270419 = 10.378125991  db
ur : 3.3012249406  = 10.3735023505 db

ETMX
ul : 0.99903400106 = -0.00839461539757 db
ll : 0.99849991349 = -0.0130393683795 db
sd : 1.00314092883 =  0.0272390056874 db
lr : 1.00046493718 =  0.00403745453682 db
ur : 1.00265600785 =  0.0230392084558 db

ETMY
ul : 1.00223179107 =  0.0193634913327 db
ll : 0.96755532811 = -0.286483823189 db
sd : 1.00861855271 =  0.0745390477589 db
lr : 1.05718545676 =  0.483023602007 db
ur : 0.99777406174 = -0.0193558045143 db
  4921   Thu Jun 30 11:36:54 2011 JamieUpdateSUSRe: ITMX whitening, ETMX left free swinging

Quote:

While closing up the whitening shop for the night, I noticed that the ITMX whitening state (Whitening "On") is opposite that of all other suspensions (they all have Whitening "Off").  I don't know which way is correct, but I assume they should all be the same.  Once all the whitening and BO testing is done, we should make sure that they're all the way we want them to be.

This was certainly my fault, probably left over from early debugging of my BO switch check script.  I've turned the ITMX whitening all off, to match the other suspensions.

Quote

Also, Koji and I are leaving ETMX free swinging.  That's the way we found it, presumably from Jamie's BO testing at the end station today.  We don't know what the optic's story is, so we're leaving it the way we found it.  Jamie (or whomever left it free swinging), can you please restore it when it is okay to do so?  Thanks!

Again, this was my fault.  Sorry.  I just accidentally left this off when I finished yesterday.  Much apologies.  I've turned the ETMX watchdog back on.

  4922   Thu Jun 30 11:40:21 2011 JamieUpdateIOORe: misc. MC work

Quote:

Jamie has started to revert the "ALTPOS" effect on the MC mirrors. So far, the screens and SLOW channels have been fixed, but the fast channels still say "ALTPOS" in the dataviewer instead of "MCL".

 The framebuilder just needed to be restarted to pull in the fixed channel names.  I restarted the framebuilder and now the channels (C1:SUS-MC2_MCL_*) are showing up properly.

  4929   Fri Jul 1 16:01:48 2011 JamieUpdateSUSETM binary whitening switching fixed

I have fixed the binary whitening switching for the ETMs (ETMX and ETMY).  See below for a description of what some of the issues were.

The ETMX whitening/no-whitening response (same measurements performed in my previous post on checking vertex sus whitening switching) looks as it should.  The ETMY response seems to indicate that the switching is happening, but the measurements are very noise.  I had to up the averaging significantly to get anything sensible.  There's something else going on with ETMY.  I'll follow up on that in another post.

response

ETMX
ul : 3.28258088774 = 10.3243087313 db
ll : 3.31203559803 = 10.4018999194 db
sd : 3.27932572306 = 10.3156911129 db
lr : 3.28189942386 = 10.3225053532 db
ur : 3.31351020008 = 10.4057662366 db

ETMY
ul : 2.9802607099  =  9.4850851468 db
ll : 1.46693103911 =  3.3281939600 db
sd : 2.19178266285 =  6.8159497462 db
lr : 2.2716636118  =  7.1268804285 db
ur : 3.42348315519 = 10.6893639064 db

End rack cable diagrams inconsistent with binary channel mapping

One of the big problems was that the most up-to-date end rack cable diagrams (that I can find) are inconsistent with the actual binary mapping. The diagram says that:

  • BO adapter chassis output A (ch 1-16)   --> CAB_1X4_26 --> cross-connect 1X4-B7 (carrying QPD whitening switching signals)
  • BO adapter chassis output B (ch 17-32) --> CAB_1X4_27 --> cross-connect 1X4-A6 (carrying OSEM whitening switching signals)

In fact, the binary outputs are switched, such that output A carries the OSEM signals, and output B carries the QPD whitening signals.

I SWITCHED THE CABLES AT THE BINARY OUTPUT ADAPTER CHASSIS so that:

  • BO adapter chassis output A (ch 1-16)   --> CAB_1X4_27 --> cross-connect 1X4-A6 (carrying OSEM whitening switching signals)
  • BO adapter chassis output B (ch 17-32) --> CAB_1X4_26 --> cross-connect 1X4-B7 (carrying QPD whitening switching signals)

The rest of the wiring remains the same.

I made the same transformation for ETMY as well.

  4930   Fri Jul 1 18:41:53 2011 JamieUpdateSUSCore optic sus damping controllers normalized

I found many of the core optic (ETMs, ITMs, BS, PRM, SRM) suspension DOF damping controllers (SUSPOS, SUSPIT, SUSYAW, SUSSIDE) to be in various states of disarray:

  • Many of the controllers did not have their "Cheby" and "BounceRoll" filters switched on.
  • Some of the controllers didn't even have the Cheby or BounceRoll filters at all, or had other different filters in their place.
  • ETMY was particularly screwy (I'll make a separate follow-up post about this)
  • A bunch of random other unused filters lying around.
  • oplev servos not on
  • etc.

I went around and tried to clean things up, by "normalizing" all of the DOF damping filter banks, ie. giving them all the same filters and clearing out unused filters, and then turning on all the appropriate filters in all core optic damping filter banks ("3:0.0", "Cheby", "BounceRoll").  I also went sure that all the outputs were properly on, and the oplev servos were on.

A couple of the optics had to have their gains adjusted to compensate for filter changes, but nothing too drastic.

Everything now looks good, and all optics are nicely damped.

I didn't touch the MC sus damping controllers, but they're in a similar state of disarray and could use a once-over as well.

 

  4931   Fri Jul 1 18:48:13 2011 JamieUpdateSUSETMY sus controller found to be in a bad state

I'm not sure what happened to ETMY SUS, but it was in a pretty bad state.  Bad burt restore, I would guess.

Most egregiously, the inputs to all of the coil output filters were switched off.  This is a bit insidious, since these inputs being off doesn't show up on the overview screen at all.  This explains why ETMY had not been damping for the last couple of day, and why my binary whitening switching measurements were nonsense.

I also found that ETMYs damping filter was a 30 Hz high pass, instead of the 3 Hz high pass in all the other suspension controllers.  Unfortunately a messed up burt restore can't explain that.

I normalized the ETMY controller to match all of the other controllers (ie. gave it a nice new 3 Hz high pass), adjusted gains accordingly, and now ETMY is behaving nicely.

  4932   Fri Jul 1 18:54:34 2011 JamieUpdateSUSETMY binary whitening switching confirmed to be fixed

After finally figuring out what was messed up with ETMY I was able to get good measurements of the binary whitening switching on ETMY to determine that it is in fact working now:

ETMY
ul : 3.2937569959  = 10.3538310999 db
ll : 3.28988426634 = 10.3436124066 db
sd : 3.34670033732 = 10.4923365497 db
lr : 3.08727050163 =  9.7914936665 db
ur : 3.27587751842 = 10.3065531117 db

  4941   Tue Jul 5 18:57:10 2011 JamieUpdateSUSMore normalization of all sus controllers

Based on Rana's comment I have gone through and moved all of the corner frequencies for the high pass filters in the SUS damping controllers to 30 Hz.  I did this for all optics (MC1, MC1, MC3, BS, ITMX, ITMY, PRM, SRM, ETMX, ETMY) all degrees of freedom (POS, PIT, YAW, SIDE).

Rana also suggested I turn off all of the BounceRoll filters until we get a chance to tune those individually for all the suspensions.

Finally, I normalized the MC SUSXXX filter banks to look just like all the other suspensions.

All damping filter banks for all degrees of freedom for all single suspensions should all be the same now (modulo the differences in the BounceRoll filters, which are now turned off).

  4944   Wed Jul 6 10:35:35 2011 JamieUpdateSUSRe : More normalization of all sus controllers

Quote:

We found the 30 Hz high pass filters had lower gain than what they used to be at low frequcnies.

So we increased the gain of the high pass filters called '30:0.0'  by a factor of 10 to have the same gain as before.

 

I'm not convinced that this is what you want to do, or at least I wouldn't do it this way.  The "k" in the zpk filter was set such that the filter had unity gain above the high-pass cut-off frequency.  For a 30 Hz high-pass the k needs to be a factor of 10 smaller than it would be for a 3 Hz high-pass to achieve this high frequency unity gain.

As it is now these HP filters have 20 dB of gain above 30 Hz.  If the open loop transfer function needs to more gain I would have done that by adjusting the overall DC gain of the filter bank, not by increasing the gain in this one filter.  Maybe you guys have been doing it differently, though.  Or maybe I'm just completely off base.

  4945   Wed Jul 6 11:45:20 2011 JamieUpdateSUSMore normalization of all sus controllers

Quote

I'm attaching a screenshot of some of the problems I see so far with MC3.

I tried to fix all of the problems that I could identify in this screen shot:

  • Fixed the TO_COIL output filter matrix screen to correctly point to the matrix element filter screens (all SUS)
  • Removed MCL sections from SUS_XXX_POSITION screens, except for MC2.  I also modified the _POSITION screens for the ETMs to refer to ALS instead of MCL.
  • Zeroed out all of the lockin gains in the TO_COIL matrices (MC SUS)
  • Made sure all whitening filter were ON (all SUS)
  • Made sure all cts2um calibration filters were on (all SUS)
  • Made sure all oplev servos were on (all SUS)
  4946   Wed Jul 6 15:32:32 2011 JamieUpdateSUSRe : More normalization of all sus controllers

So after talking to Kiwamu about it, I understand now that since the damping loops need all of this extra gain when the high-pass corner is moved up, it's more convenient to put that gain in the control filter itself, rather than having to crank the overall DC gain up to some inconveniently high value.

  4961   Tue Jul 12 10:18:05 2011 JamieUpdateCDSC1:DAQ-FB0_C1???_STATUS indicators red, restored after controller restarts

Yesterday I found the C1:DAQ-FB0_C1???_STATUS lights to be red for the SUS, MCS, SCX, and SCY controllers.  I know this has something to do with model communication with the framebuilder, but I unfortunately don't remember exactly what it is.  I decided to try restarting the affected models to see if that cleared up the problem.  It did.  After restarting c1scx, c1scy, c1sus, and c1mcs everything came back up green.

We need some better documentation about what all of these status indicators mean.

  4971   Fri Jul 15 08:48:36 2011 JamieSummarySUSPhotosensor Head Lessons

Nicole: I thought we had decided to use teflon as the insulator between the PCB (yellow) and the LED/PDs?  I don't think you should use another circuit board with copper on it.  The copper will short the LED/PD heads to the metal box, which might be problematic.

Otherwise the design looks pretty good.  I think the PDs have three leads each, yes?

  5005   Wed Jul 20 19:48:03 2011 JamieUpdateSUSRe: oplev gains today

 

 We have been modifying models that need to have their channels renamed to run activateDQ when Joe's post_build_script to is run. 

The trick is to integrate things to get the post_build_script running after every model build (getting it hooked in to the make file somehow).  We're working on it.

 I've added the following epics channels to sus_single_control model using the epicsOutput part:

  • OL_SUM
  • OL_PITCH
  • OL_YAW

These channels are now all available.  I'm not exactly sure how to ensure that they're being trended.  I'll check that tomorrow.

  5006   Wed Jul 20 20:04:54 2011 JamieUpdateCDSC1:DAQ-FB0_C1XXX_STATUS sometimes unexplainably goes red

I have been noticing this happening occasionally, but I don't understand what is causing:

status-fb-red1.png

The channel in question above is C1:DAQ-FB0_C1SCX_STATUS.  This channel is (I believe) reporting some status of the front end model communication with the frame builder, but I'm not sure exactly what.

Usually this problem goes away when I restart the model or the frame builder, but it didn't work this time.  Tomorrow I will figure out what this channel means, why it's sporadically going red, and how to correct it.

  5007   Wed Jul 20 20:44:56 2011 JamieUpdateSUSAll sus models rebuilt and restarted

There were a couple of recent improvements to the sus_single_control model that had not been propagated to all of the suspension controllers.

Rebuilt and restarted c1mcs, c1sus, c1scx, and c1scy.  Everything seems to be working fine after restart.

  5031   Mon Jul 25 13:09:39 2011 JamieUpdateCDSc1ioo Make problem

> It looks like something wrong with Makefile.

Sorry, this was my bad.  I was making a patch to the makefile to submit back upstream and I forgot to revert my changes.  I've reverted them now, so everything should be back to normal.

  5032   Mon Jul 25 17:16:02 2011 JamieUpdateSUSNow acquiring SUSXXX_IN1_DQ channels

> And.....we have also lost the DAQ channels that used to be associated with the _IN1 of the SUSPOS/PIT/YAW filter modules. Please put them back; our templates don't work without them.

I have (re?)added the SUS{POS,PIT,YAW,SIDE}_IN1_DQ channels.  I did this by modifying the activateDQ.py script to always turn them on [0].  They should now always be activated after the activateDQ script is run.

[0] This script now lives in the cds_user_apps repo at cds/c1/scripts/activateDQ.py

 

  6112   Tue Dec 13 11:51:33 2011 JamieUpdateComputersDid someone just do something to fb??

Quote:

Dataviewer couldn't connect to the framebuilder, so I checked the CDS status screen, and all the fb-related things on each model went white, then red, then computer-by-computer they came back green.  Now dataviewer works again.  Is someone secretly doing shit while not in the lab???  Not cool man!

This happens on occasion, and I have reported it to the CDS guys.  Something apparently causes the framebuilder to crash, but I haven't figured out what it is yet.  I doubt this particular instance had anything to do with remote futzing.

  6159   Tue Jan 3 15:49:27 2012 JamieUpdateComputerspossible front-end timing issue

Quote:

Is there a reason the framebuilder status light is red for all the front ends?

Also, I reenabled PRM watchdog.

Apparently there is a bug in the timing cards having to do with the new year roll-over that is causing front-end problems.  From Rolf:

For systems using the Spectracom IRIG-B cards for timing information, the code did not properly roll over the time for
2012 (still thinks it is 2011 and get reports from DAQ of timing errors (0x4000)). I have made a temporary fix for this
in the controller.c code in branch-2.3, branch-2.4 and release 2.3.1. 

I was going to check to see if the 40m is suffering from this. I'll be over to see if that's the problem.

  6169   Wed Jan 4 11:47:08 2012 JamieSummaryComputer Scripts / Programsmedm directory clean-up

Quote:

 I moved the following old and obsolete MEDM directories to an archive /cvs/cds/rtcds/caltech/c1/medm

  • c1vga
  • c1vgl
  • c1gpt
  • c1gpv
  • c1nio
  • c1spy
  • c1spx

None of these models were present in /cvs/cds/rtcds/caltech/c1/target

Remember that we don't use /cvs/cds/rtcds anymore.  Everything should be in /opt/rtcds now.

  6171   Wed Jan 4 16:40:52 2012 JamieUpdateComputersfront-end fb communication restored

Communication between the front end models and the framebuilder has been restored.  I'm not sure exactly what the issue was, but rebuilding the framebuilder daqd executable and restarting seems to have fixed the issue.

I suspect that the problem might have had to do with how I left things after the last attempt to upgrade to RCG 2.4.  Maybe the daqd that was running was linked against some library that I accidentally moved after starting the daqd process.  It would have kept running fine as was, but if the process died and was attempted to be started again, it's broken linking might have kept it from running correctly.  I don't have any other explanation.

It turns out this was not (best I can tell) related to the new year time sync issues that wer seen at the sites.

  6173   Thu Jan 5 09:59:27 2012 JamieUpdateCDSRTS/RCG/DAQ UPGRADE TO COMMENCE

RTS/RCG/DAQ UPGRADE TO COMMENCE

I will be attempting (again) to upgrade the RTS, including the RCG and the daqd, to version 2.4 today.  The RTS will be offline until further notice.

  6174   Thu Jan 5 20:40:21 2012 JamieUpdateCDSRTS upgrade aborted; restored to previous settings; fb symmetricom card failing?

After running into more problems with the upgrade, I eventually decided to abort todays upgrade attempt, and revert back to where we were this morning (RTS 2.1).  I'll try to follow this with a fuller report explaining what problems I encountered when attempting the upgrade.

However, when Alex and I were trying to figure out what was going wrong in the upgrade, it appears that the fb symmetricom card lost the ability to sync with the GPS receiver.  When the symmeticom module is loaded, dmesg shows the following:

[  285.591880] Symmetricom GPS card on bus 6; device 0
[  285.591887] PIC BASE 2 address = fc1ff800
[  285.591924] Remapped 0x17e2800
[  285.591932] Current time 947125171s 94264us 800ns 
[  285.591940] Current time 947125171s 94272us 600ns 
[  285.591947] Current time 947125171s 94280us 200ns 
[  285.591955] Current time 947125171s 94287us 700ns 
[  285.591963] Current time 947125171s 94295us 800ns 
[  285.591970] Current time 947125171s 94303us 300ns 
[  285.591978] Current time 947125171s 94310us 800ns 
[  285.591985] Current time 947125171s 94318us 300ns 
[  285.591993] Current time 947125171s 94325us 800ns 
[  285.592001] Current time 947125171s 94333us 900ns 
[  285.592005] Flywheeling, unlocked...

Because of this, the daqd doesn't get the proper timing signal, and consequently is out of sync with the timing from the models.

It's completely unclear what caused this to happen.  The card seemed to be working all day today, then Alex and I were trying to debug some other(maybe?) timing issues and the symmetricom card all of a sudden stopped syncing to the GPS.  We tried rebooting the frame builder and even tried pulling all the power to the machine, but it never came back up.  We checked the GPS signal itself and to the extend that we know what that signal is supposed to look like it looked ok.

I speculate that this is also the cause of the problems were were seeing earlier in the week.  Maybe the symmetricom card has just been acting flaky, and something we did pushed it over the edge.

Anyway, we will try to replace it tomorrow, but Alex is skeptical that we have a replacement of this same card.  There may be a newer Spectracom card we can use, but there may be problems using it on the old sun hardware that the fb is currently running on.  We'll see.

In the mean time, the daqd is running rogue, off of it's own timing.  Surprisingly all of the models are currently showing 0x0 status, which means no problems.  It doesn't seem to be recording any data, though.  Hopefully we'll get it all sorted out tomorrow.

  6176   Fri Jan 6 11:49:13 2012 JamieUpdateCDSframebuilder taken offline to diagnose problem with symmetricom timing card

Alex and I have taken the framebuilder offline to try to see what's wrong with the symmetricom card.  We have removed the card from the chassis and Alex has taken it back to downs to do some more debugging.

We have been formulating some alternate methods to get timing to the fb in case we can't end up getting the card working.

  6177   Fri Jan 6 14:31:54 2012 JamieUpdateCDSframebuilder back online, using NTP time syncronization

The framebuilder is back online now, minus it's symmetricom GPS card.  The card seems to have failed entirely, and was not able to be made to work at downs either.  It has been entirely removed from fb.

As a fall back, the system has been made to work off of the system NTP-based time synchronization.  The latest symmetricom driver, which is part of the RCG 2.4 branch, will fall back to using local time if the GPS synchronization fails.  The new driver was compiled from our local checkout of the 2.4 source in the new to-be-used-in-the-future rtscore directory:

controls@fb ~ 0$ diff {/opt/rtcds/rtscore/branches/branch-2.4/src/drv/symmetricom,/lib/modules/2.6.34.1/kernel/drivers/symmetricom}/symmetricom.ko
controls@fb ~ 0$ 

The driver was reloaded.   daqd was also linked against the last running stable version and restarted:

controls@fb ~ 0$ ls -al $(readlink -f /opt/rtcds/caltech/c1/target/fb/daqd)
-rwxr-xr-x 1 controls controls 6592694 Dec 15 21:09 /opt/rtcds/caltech/c1/target/fb/daqd.20120104
controls@fb ~ 0$ 

We'll have to keep an eye on the system, to see that it continues to record data properly, and that the fb and the front-ends remain in sync.

The question now is what do we do moving forward.  CDS is not supporting the symmetricom cards anymore, and have moved to using Spectracom GPS/IRIG-B cards.  However, Downs has neither at the moment.  Even if we get a new Spectracom card, it might not work in this older Sun hardware, in which case we might need to consider upgrading the framebuilder to a new machine (one supported by CDS).

  6305   Wed Feb 22 16:55:16 2012 JamieUpdateSUSwacky state of SUS input matrices

While Kiwamu and I were trying to investigate the the vertex glitches we were noticing excess noise in ITMX, which Kiwamu blamed on some sort of bad diagonalization.  Sure enough, the ITMX input matrix is in the default state [0], not a properly diagonalized state.  Looking through the rest of the suspensions, I found PRM also in the default state, not diagonalized.

We should do another round of suspension diagonalization.

Kiwamu (or whoever is here last tonight): please run the free-swing/kick script (/opt/rtcds/caltech/c1/scripts/SUS/freeswing) before you leave, and I'll check the matrices and update the suspensions tomorrow morning.

[0]

0.25 0.25 0.25 0.25 0
1.66 1.66 -1.66 1.66 0
1.66 -1.66 -1.66 1.66 0
0 0 0 0 1

  6467   Thu Mar 29 19:13:56 2012 JamieOmnistructureComputersWireless router for GC

I retrieved the newly "secured" router from Junaid.  It had apparently been hooked up to the GC network via it's LAN port, but it's LAN services had no been shut off.  It was therefore offering a competing DHCP server, which will totally hose a network.  A definite NONO.

The new SSID is "40mWiFi", it's WPA2, and the password is pasted to the bottom of the unit (the unit is back in it's original spot on the office computer rack.

  6495   Fri Apr 6 14:39:21 2012 JamieUpdateComputersRAID array is rebuilding....

The RAID (JetStor SATA 416S) is indeed resyncing itself after a disk failure.  There is a hot spare, so it's stable for the moment.  But we need a replacement disk:

    RAID disks:  1000.2GB Hitachi HDT721010SLA360

Do we have spares?  If not we should probably buy some, if we can.  We want to try to keep a stock of the same model number.

Other notes:

The RAID has a web interface, but it was for some reason not connected.  I connected it to the martian network at 192.168.113.119.

Viewing the RAID event log on the web interface silences the alarm.

I retrieved the manual from Alex, and placed it in the COMPUTER MANUALS drawer in the filing cabinet.

  6501   Fri Apr 6 20:05:12 2012 JamieSummaryGeneralLaser Emergency Shutoff

We reset the interlock and restarted the PSL.  The end AUX lasers seem to have come back online fine.  PMC and mode cleaner locked back up quickly.

  6540   Tue Apr 17 11:05:04 2012 JamieUpdateCDSCDS upgrade in progress

I am continuing to attempt to upgrade the CDS system to RTS 2.5.  Systems will continue to be up and down for the rest of the day.

  6541   Tue Apr 17 19:03:09 2012 JamieUpdateCDSCDS upgrade in progress

Upgrade progresses, but not complete.  There are some relatively minor issues, and one potentially big issue.

All new software has been installed, including the new epics that supports long channel names.

I've been doing a LOT of cleanup.  It was REALLY messy in there.

The new framebuilder/daqd code is running on fb.

Models are compiling with the new RCG and I am able to get them running.  Some of them are not compiling for relatively minor reasons (the simulink models need updating).  I'm also running into compile problems with IOPs that are using the dolphin drivers.

The major issue is that the framebuilder and the models are not syncing their timing, so there's no data collection.  I've spoken to Alex and he and Rolf are going to come over tomorrow to sort it out.  It's possible that we're missing timing hardware that the new code is expecting.

There are still some stability issues I haven't sorted out yet, and I have a lot more cleanup to do.

At this rate I'm going to shoot for being done Thursday.

  6542   Wed Apr 18 08:53:50 2012 JamieUpdateGeneralPower outage last night

Apparently there was a catastrophic power failure last night.  Bob says it took out power in most of Pasadena.

Bob checked the vacuum system when he got in first thing this morning and everything's back up and checks out.  The laster is still off and most of the front-end computers did not recover.

I'm going to start a boot fest now.  I'll be able to report more once everything is back on.

  6543   Wed Apr 18 10:05:40 2012 JamieUpdateGeneralPower outage last night

All of the front-ends are back up and I've been able to recover local control of all of the opitcs (snapshots from saturday). Issues:

  • I can't lock the PMC.  Still unclear why.
  • there are no oplev signals in MC1, MC2, and MC3
  • Something is wrong with PRM.  He is very noisy.  Turning on his oplev servo makes him go crazy.
  • There are all sorts of problems with the oplevs in general.  Many of the optics have no oplev settings.  This is probably not related to the power outage.

On a brighter note, ETMX is damped with it's new RCG 2.5 controller!  yay!

  6546   Wed Apr 18 19:59:48 2012 JamieUpdateCDSCDS upgrade success

The upgrade is nearly complete:

  • new daqd code is running on fb
  • the fe/daqd timing issue was resolved by adjusting the GPS offset in the daqdrc.  I will document this more later.
  • the power outage conveniently rebooted all the front-end machines, so they're all now running new caRepeater
  • all models have been successfully recompiled with RCG 2.5 (with only a couple small glitches)
  • all new models are running on all front-end machines (with a couple exceptions)
  • all suspension models seem to be damping under local control (PRM is having troubles that are likely unrelated to the upgrade).
  • a lot of cleanup has been done

Remaining tasks/issues:

  • more testing OF EVERYTHING needs o be done
  • I did not yet update the DIS dolphin code, so we're running with the old code.  I don't think this is a problem, but it would be nice to get us running what they're running at the sites
  • I tried to cleanup/simplify how front-end initialization is done.  However, there is a problem and models are not auto-starting after reboot.  This needs to be fixed.
  • the userapps directory is in a new place (/opt/rtcds/userapps).  Not everything in the old location was checked into the repository, so we need to check to make sure everything that needs to be is checked in, and that all the models are running the right code.
  • the c1oaf model seems to be having a dolphin issue that needs to be sorted
  • the c1gfd model causes c1ioo to crash immediately upon being loaded.  I have removed it from the rtsystab.  That model needs to be fixed.
  • general model cleanup is in order.
  • more front-end cleanup is needed, particularly in regards to boot-up procedure.
  • document the entire upgrade procedure.

I'll finish up these remaining tasks tomorrow.

  6548   Thu Apr 19 08:43:16 2012 JamieUpdateCDSoaf

Quote:

Edit: Old version (~september) of the code and oaf model is running now. In the 2.1 code there was a link from src/epics/simLink to oaf code for each DOF. It seems that 2.5 version finds models and c codes in standard directories. I need to move working code to the proper directory.

 Yes, things have changed.  Please wait until I have things cleaned up before working on models.  I'll explain what the new setup is.

  6552   Fri Apr 20 19:54:57 2012 JamieUpdateCDSCDS upgrade problems

I ran into a couple of snags today.

A big one is that the framebuilder daqd started going haywire when I told it to start writing frames.  After restart the logs started showing this:

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

and the network seemed like it started to get really slow.  I wasn't able to figure out what was going on, so I shut the frame writing off again.  I'll have to work with Rolf on that next week.

Another big problem is the workstation application upgrades.  The NDS protocol version has been incremented, which means that all the NDS client applications have to be upgraded.  The new dataviewer is working fine (on pianosa), but dtt is not:

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$ 

I don't know what's going on here.  All the library paths are ok.  Hopefully I'll be able to figure this out soon.  The old version of dtt definitely does not work with the new setup.

I might go ahead and upgrade some more of the workstations to Ubuntu in the next couple of days as well, so everything is more on the same page.

I also tried to cleanup the front-end boot process, which has it's own problems (models won't auto-start).  I haven't figured that out yet either.  It really needs to just be completely overhauled.

  6554   Sat Apr 21 17:38:19 2012 JamieUpdateCDSdtt, dataviewer working; problem with trend frames

Quote:

Another big problem is the workstation application upgrades.  The NDS protocol version has been incremented, which means that all the NDS client applications have to be upgraded.  The new dataviewer is working fine (on pianosa), but dtt is not:

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$

 dtt (diaggui) and dataviewer are now working on pianosa to retrieve realtime data and past data from DQ channels.

Unfortunately it looks like there may be a problem with trend data, though.  If I try to retrieve 1 minute of "full data" with dataviewer for channel C1:SUS-ITMX_SUSPOS_IN1_DQ around GPS 1019089138 everything works fine:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

but if I specify any trend data (second, minute, etc.) I get the following:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex warned me that this might have happened when I was trying to test the new daqd without first turning off frame writing.

I'm not sure how to check the integrity of the frames, though.  Hopefully they can help sort this out on Monday.

  6555   Sat Apr 21 20:40:28 2012 JamieUpdateCDSframebuilder frame writing working again

Quote:

A big one is that the framebuilder daqd started going haywire when I told it to start writing frames.  After restart the logs started showing this:

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

and the network seemed like it started to get really slow.  I wasn't able to figure out what was going on, so I shut the frame writing off again.  I'll have to work with Rolf on that next week.

So the framebuilder/daqd frame writing issue seems to have been a transient one.  Alex said he tried enabling frame writing manually and it worked fine, so I tried re-enabling the frame writing config lines and sure enough it worked fine.  So it's a mystery.  Alex said the "main profiler warning" lines tend to show up when data is backed up in the buffer, although he didn't explain why exactly that would have been the issue here.

daqdrc frame writing directives:

start frame-saver;
sync frame-saver;
start trender;
start trend-frame-saver;
sync trend-frame-saver;
start minute-trend-frame-saver;
sync minute-trend-frame-saver;
start raw_minute_trend_saver;
  6556   Sat Apr 21 21:10:34 2012 JamieUpdateCDStrend frame issue partially resolved

Quote:

Unfortunately it looks like there may be a problem with trend data, though.  If I try to retrieve 1 minute of "full data" with dataviewer for channel C1:SUS-ITMX_SUSPOS_IN1_DQ around GPS 1019089138 everything works fine:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

but if I specify any trend data (second, minute, etc.) I get the following:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex warned me that this might have happened when I was trying to test the new daqd without first turning off frame writing.

Alex told me that the "trend data is not available" message comes from the "trender" functionality not being enabled in daqd.  After re-enabling it (see #6555) minute trend data was available again.  However, there still seems to be an issue with second trends.  When I try to retrieve second trend data from dataviewer for which minute trend data *is* available I get the following error message:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

Awaiting more help from Alex...

  6560   Tue Apr 24 14:30:08 2012 JamieUpdateCDSIntroducing: rtcds, a utility for interacting with the CDS RTS/RCG

The new rtcds utility

I have written a new utility for interacting with the CDS RTS/RCG system: "rtcds".  It should be available on all workstations and front-end machines, but certain commands are restricted to run on certain front end machines (build, start, stop, etc.).  Here's the help:

controls@c1lsc ~ 0$ rtcds help
Usage: rtcds <command> [arg]

Available commands:

  build|make SYS      build model
  install SYS         install model
  start SYS|all       start model
  restart SYS|all     restart running model
  stop|kill SYS|all   stop running model
  list                list all models for current host

controls@c1lsc ~ 0$ 

Please use this new utility from now on when interacting with RTS.

I'll be improving and expanding it as we go.   Please let me know if you encounter any problems.

 

  6561   Tue Apr 24 14:35:37 2012 JamieUpdateCDSlimited second trend lookback

Quote:

Alex told me that the "trend data is not available" message comes from the "trender" functionality not being enabled in daqd.  After re-enabling it (see #6555) minute trend data was available again.  However, there still seems to be an issue with second trends.  When I try to retrieve second trend data from dataviewer for which minute trend data *is* available I get the following error message:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

Awaiting more help from Alex...

It looks like this is actually just a limit of how long we're saving the second trends, which is just not that long.  I'll look into extending the second trend look-back.

ELOG V3.1.3-