In order to install the BO module in 1X2, I need to shut down all DC power to the 1X1 and 1X2 racks.

All power has been restored to the 1X1 and 1X2 racks.  The modecleaner is locked again.

I have also hooked up the binary output module in 1X2, which was never actually powered.  This controls the whitening filters for MC WFS.  Still needs to be tested.

Kiwamu and I have started to put together a vent plan on the 40m wiki:

http://blue.ligo-wa.caltech.edu:8000/40m/vent/201107

We will keep working on this (there's still a *lot* to fill in), but please help fill in the plan by adding questions, answers, procedures, preparations, etc.

I recorded a burt snapshot of these settings: /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2011/Jun/23/21:40

I have updated the sus_single_control model, adjusting/cleaning up/simplifying the LSC/POS input signals, and routing new signals to the lockins. Notably one of POS inputs to the part ("lockin_in") was eliminated (see below).

The 6 inputs to the TO_COIL output matrix are now:

LSCPOS + OFFSET + ALT_POS_IN
ASCPIT + OFFSET + SUSPIT + OLPIT
ASCYAW +OFFSET + SUSYAW + OLYAW
SIDE
LOCKIN1
LOCKIN2

The ALT_POS input is used only by the ETMs for the green locking. Just outside of the sus_single_control library part in the ETM models are the green locking controls, consisting of the ETM?_ALS filter bank and the ETM?_GLOCKIN lockin, the outputs from which are summed and fed into the aforementioned ALT_POS input.

As for the SUS lockins (LOCKIN1 and LOCKIN2 in the library model), their input matrix now gets the direct inputs from the OSEMS (before filtering) and the outputs to the coils, after all filtering. These will aid in doing binary output switching tests.

All suspension models (c1sus, c1scx, c1scy) have been rebuild and restarted so that they reflect these changes.

IMPORTANT ELECTRONICS SAFETY LESSON TO FOLLOW

Yesterday, I fried the +15 V power supply rail on one of the Eurocrate extender cards while I was checking out the binary switching in the 1X5 rack.  I will describe what I did it in the hopes that everyone else will be less stupid than me.

I wanted to monitor the voltage across a resistor on the suspension OSEM whitening board.  Since I knew that both sides of the resistor would be at non-zero voltage (including possibly at the power-supply rail), I used a battery-operated scope with floating inputs, so that the scope would not try to pull the probe shield to ground.  That should be OK, although not recommended, as you'll see, because you must be very careful to make sure that the scopes inputs are indeed floating.

Let's call the original signal 'A'.  The trouble came when I then connected another signal (B), whose shield was connected to the ground on the whitening board, to the scope.  Apparently the grounds on the scope inputs are connected, or were in the configuration I was using.  When I connected the signal B, B's ground shorted A's shield to ground, which had been sitting at the +15V rail.  That short circuit then fried the +15V supply line on the extender card I was using (escaping magic smoke was detected).  Thankfully this only blew the extender card, and not the Eurocrate or the Eurocrate power supply or the whitening board or the scope etc, all of which would have been much worse.

The moral of the story is to be very careful when connecting power supply voltages to the shield or ground of a scope.  In short, don't do it.  I didn't ultimately need to, since I could have found other ways to measure the same signal.

I have been checking the binary output switching for the SUS whitening filters. It appears that the whitening switching is working for (almost) all the vertex suspensions (BS, ITMX, ITMY, PRM, SRM), but not for the ETMs.

The table below lists the output from my switch-checking script (attached). The script uses the SUS digital lockin to drive one coil and measure the same coil's OSEM response, repeating for each coil/OSEM pair. I used a lockin drive frequency of about 10 Hz, at which the whitening filter should have 10 db of gain.

All but one of the vertex OSEMS show the proper response (~10db gain at 10Hz) when the whitening is switched on from the digital controls. ITMY UL appears to not be switching, which I fear is due to my electronics fail noted in my previous log post.  The ETMs are clearly not switching at all.

I will try to get the ETM switching working tomorrow, as well as try to asses what can be done about the ITMY UL switch.  After that I will work on confirming the coil drive dewhite switching.

lockin settings

freq: 10.123 Hz
amp: 10000
I/Q filters: 0.1 Hz LP, 4-pole butterworth

response

BS
ul : 3.31084503062 = 10.3987770676 db
ll : 3.34162124753 = 10.4791444741 db
sd : 3.43226254574 = 10.7116100229 db
lr : 3.28602651913 = 10.3334212798 db
ur : 3.29361593249 = 10.3534590969 db

ITMX
ul : 3.37499773336 = 10.5654697099 db
ll : 3.2760924572  = 10.3071229966 db
sd : 3.13374799272 =  9.9212813757 db
lr : 3.28133776018 = 10.3210187243 db
ur : 3.37250879937 = 10.5590618297 db

ITMY
ul : 0.99486434364 = -0.0447226830807 db
ll : 3.39420873724 = 10.6147709414 db
sd : 3.88698713176 = 11.7922620572 db
lr : 3.357123865   = 10.5193473069 db
ur : 3.37876008179 = 10.5751470918 db

PRM
ul : 3.26758918055 = 10.2845489876 db
ll : 3.32023820566 = 10.4233848529 db
sd : 3.25205538857 = 10.2431586766 db
lr : 3.24610681962 = 10.227256141  db
ur : 3.31311970305 = 10.4047425446 db

SRM
ul : 3.30506423619 = 10.3835980943 db
ll : 3.28152094133 = 10.3215036019 db
sd : 3.08566647696 =  9.7869796462 db
lr : 3.30298270419 = 10.378125991  db
ur : 3.3012249406  = 10.3735023505 db

ETMX
ul : 0.99903400106 = -0.00839461539757 db
ll : 0.99849991349 = -0.0130393683795 db
sd : 1.00314092883 =  0.0272390056874 db
lr : 1.00046493718 =  0.00403745453682 db
ur : 1.00265600785 =  0.0230392084558 db

ETMY
ul : 1.00223179107 =  0.0193634913327 db
ll : 0.96755532811 = -0.286483823189 db
sd : 1.00861855271 =  0.0745390477589 db
lr : 1.05718545676 =  0.483023602007 db
ur : 0.99777406174 = -0.0193558045143 db

This was certainly my fault, probably left over from early debugging of my BO switch check script.  I've turned the ITMX whitening all off, to match the other suspensions.

Again, this was my fault.  Sorry.  I just accidentally left this off when I finished yesterday.  Much apologies.  I've turned the ETMX watchdog back on.

 The framebuilder just needed to be restarted to pull in the fixed channel names.  I restarted the framebuilder and now the channels (C1:SUS-MC2_MCL_*) are showing up properly.

I have fixed the binary whitening switching for the ETMs (ETMX and ETMY).  See below for a description of what some of the issues were.

The ETMX whitening/no-whitening response (same measurements performed in my previous post on checking vertex sus whitening switching) looks as it should.  The ETMY response seems to indicate that the switching is happening, but the measurements are very noise.  I had to up the averaging significantly to get anything sensible.  There's something else going on with ETMY.  I'll follow up on that in another post.

response

ETMX
ul : 3.28258088774 = 10.3243087313 db
ll : 3.31203559803 = 10.4018999194 db
sd : 3.27932572306 = 10.3156911129 db
lr : 3.28189942386 = 10.3225053532 db
ur : 3.31351020008 = 10.4057662366 db

ETMY
ul : 2.9802607099  =  9.4850851468 db
ll : 1.46693103911 =  3.3281939600 db
sd : 2.19178266285 =  6.8159497462 db
lr : 2.2716636118  =  7.1268804285 db
ur : 3.42348315519 = 10.6893639064 db

End rack cable diagrams inconsistent with binary channel mapping

One of the big problems was that the most up-to-date end rack cable diagrams (that I can find) are inconsistent with the actual binary mapping. The diagram says that:

• BO adapter chassis output A (ch 1-16)   --> CAB_1X4_26 --> cross-connect 1X4-B7 (carrying QPD whitening switching signals)
• BO adapter chassis output B (ch 17-32) --> CAB_1X4_27 --> cross-connect 1X4-A6 (carrying OSEM whitening switching signals)

In fact, the binary outputs are switched, such that output A carries the OSEM signals, and output B carries the QPD whitening signals.

I SWITCHED THE CABLES AT THE BINARY OUTPUT ADAPTER CHASSIS so that:

• BO adapter chassis output A (ch 1-16)   --> CAB_1X4_27 --> cross-connect 1X4-A6 (carrying OSEM whitening switching signals)
• BO adapter chassis output B (ch 17-32) --> CAB_1X4_26 --> cross-connect 1X4-B7 (carrying QPD whitening switching signals)

The rest of the wiring remains the same.

I made the same transformation for ETMY as well.

I found many of the core optic (ETMs, ITMs, BS, PRM, SRM) suspension DOF damping controllers (SUSPOS, SUSPIT, SUSYAW, SUSSIDE) to be in various states of disarray:

• Many of the controllers did not have their "Cheby" and "BounceRoll" filters switched on.
• Some of the controllers didn't even have the Cheby or BounceRoll filters at all, or had other different filters in their place.
• ETMY was particularly screwy (I'll make a separate follow-up post about this)
• A bunch of random other unused filters lying around.
• oplev servos not on
• etc.

I went around and tried to clean things up, by "normalizing" all of the DOF damping filter banks, ie. giving them all the same filters and clearing out unused filters, and then turning on all the appropriate filters in all core optic damping filter banks ("3:0.0", "Cheby", "BounceRoll").  I also went sure that all the outputs were properly on, and the oplev servos were on.

A couple of the optics had to have their gains adjusted to compensate for filter changes, but nothing too drastic.

Everything now looks good, and all optics are nicely damped.

I didn't touch the MC sus damping controllers, but they're in a similar state of disarray and could use a once-over as well.

I'm not sure what happened to ETMY SUS, but it was in a pretty bad state.  Bad burt restore, I would guess.

Most egregiously, the inputs to all of the coil output filters were switched off.  This is a bit insidious, since these inputs being off doesn't show up on the overview screen at all.  This explains why ETMY had not been damping for the last couple of day, and why my binary whitening switching measurements were nonsense.

I also found that ETMYs damping filter was a 30 Hz high pass, instead of the 3 Hz high pass in all the other suspension controllers.  Unfortunately a messed up burt restore can't explain that.

I normalized the ETMY controller to match all of the other controllers (ie. gave it a nice new 3 Hz high pass), adjusted gains accordingly, and now ETMY is behaving nicely.

After finally figuring out what was messed up with ETMY I was able to get good measurements of the binary whitening switching on ETMY to determine that it is in fact working now:

ETMY
ul : 3.2937569959  = 10.3538310999 db
ll : 3.28988426634 = 10.3436124066 db
sd : 3.34670033732 = 10.4923365497 db
lr : 3.08727050163 =  9.7914936665 db
ur : 3.27587751842 = 10.3065531117 db

Based on Rana's comment I have gone through and moved all of the corner frequencies for the high pass filters in the SUS damping controllers to 30 Hz.  I did this for all optics (MC1, MC1, MC3, BS, ITMX, ITMY, PRM, SRM, ETMX, ETMY) all degrees of freedom (POS, PIT, YAW, SIDE).

Rana also suggested I turn off all of the BounceRoll filters until we get a chance to tune those individually for all the suspensions.

Finally, I normalized the MC SUSXXX filter banks to look just like all the other suspensions.

All damping filter banks for all degrees of freedom for all single suspensions should all be the same now (modulo the differences in the BounceRoll filters, which are now turned off).

I'm not convinced that this is what you want to do, or at least I wouldn't do it this way.  The "k" in the zpk filter was set such that the filter had unity gain above the high-pass cut-off frequency.  For a 30 Hz high-pass the k needs to be a factor of 10 smaller than it would be for a 3 Hz high-pass to achieve this high frequency unity gain.

As it is now these HP filters have 20 dB of gain above 30 Hz.  If the open loop transfer function needs to more gain I would have done that by adjusting the overall DC gain of the filter bank, not by increasing the gain in this one filter.  Maybe you guys have been doing it differently, though.  Or maybe I'm just completely off base.

I tried to fix all of the problems that I could identify in this screen shot:

• Fixed the TO_COIL output filter matrix screen to correctly point to the matrix element filter screens (all SUS)
• Removed MCL sections from SUS_XXX_POSITION screens, except for MC2.  I also modified the _POSITION screens for the ETMs to refer to ALS instead of MCL.
• Zeroed out all of the lockin gains in the TO_COIL matrices (MC SUS)
• Made sure all whitening filter were ON (all SUS)
• Made sure all cts2um calibration filters were on (all SUS)
• Made sure all oplev servos were on (all SUS)

So after talking to Kiwamu about it, I understand now that since the damping loops need all of this extra gain when the high-pass corner is moved up, it's more convenient to put that gain in the control filter itself, rather than having to crank the overall DC gain up to some inconveniently high value.

Yesterday I found the C1:DAQ-FB0_C1???_STATUS lights to be red for the SUS, MCS, SCX, and SCY controllers.  I know this has something to do with model communication with the framebuilder, but I unfortunately don't remember exactly what it is.  I decided to try restarting the affected models to see if that cleared up the problem.  It did.  After restarting c1scx, c1scy, c1sus, and c1mcs everything came back up green.

We need some better documentation about what all of these status indicators mean.

Nicole: I thought we had decided to use teflon as the insulator between the PCB (yellow) and the LED/PDs?  I don't think you should use another circuit board with copper on it.  The copper will short the LED/PD heads to the metal box, which might be problematic.

Otherwise the design looks pretty good.  I think the PDs have three leads each, yes?

 We have been modifying models that need to have their channels renamed to run activateDQ when Joe's post_build_script to is run. 

The trick is to integrate things to get the post_build_script running after every model build (getting it hooked in to the make file somehow).  We're working on it.

 I've added the following epics channels to sus_single_control model using the epicsOutput part:

• OL_SUM
• OL_PITCH
• OL_YAW

These channels are now all available.  I'm not exactly sure how to ensure that they're being trended.  I'll check that tomorrow.

I have been noticing this happening occasionally, but I don't understand what is causing:

The channel in question above is C1:DAQ-FB0_C1SCX_STATUS.  This channel is (I believe) reporting some status of the front end model communication with the frame builder, but I'm not sure exactly what.

Usually this problem goes away when I restart the model or the frame builder, but it didn't work this time.  Tomorrow I will figure out what this channel means, why it's sporadically going red, and how to correct it.

There were a couple of recent improvements to the sus_single_control model that had not been propagated to all of the suspension controllers.

Rebuilt and restarted c1mcs, c1sus, c1scx, and c1scy.  Everything seems to be working fine after restart.

> It looks like something wrong with Makefile.

Sorry, this was my bad.  I was making a patch to the makefile to submit back upstream and I forgot to revert my changes.  I've reverted them now, so everything should be back to normal.

> And.....we have also lost the DAQ channels that used to be associated with the _IN1 of the SUSPOS/PIT/YAW filter modules. Please put them back; our templates don't work without them.

I have (re?)added the SUS{POS,PIT,YAW,SIDE}_IN1_DQ channels.  I did this by modifying the activateDQ.py script to always turn them on [0].  They should now always be activated after the activateDQ script is run.

[0] This script now lives in the cds_user_apps repo at cds/c1/scripts/activateDQ.py

This happens on occasion, and I have reported it to the CDS guys.  Something apparently causes the framebuilder to crash, but I haven't figured out what it is yet.  I doubt this particular instance had anything to do with remote futzing.

Apparently there is a bug in the timing cards having to do with the new year roll-over that is causing front-end problems.  From Rolf:

For systems using the Spectracom IRIG-B cards for timing information, the code did not properly roll over the time for
2012 (still thinks it is 2011 and get reports from DAQ of timing errors (0x4000)). I have made a temporary fix for this
in the controller.c code in branch-2.3, branch-2.4 and release 2.3.1. 

I was going to check to see if the 40m is suffering from this. I'll be over to see if that's the problem.

Remember that we don't use /cvs/cds/rtcds anymore.  Everything should be in /opt/rtcds now.

Communication between the front end models and the framebuilder has been restored.  I'm not sure exactly what the issue was, but rebuilding the framebuilder daqd executable and restarting seems to have fixed the issue.

I suspect that the problem might have had to do with how I left things after the last attempt to upgrade to RCG 2.4.  Maybe the daqd that was running was linked against some library that I accidentally moved after starting the daqd process.  It would have kept running fine as was, but if the process died and was attempted to be started again, it's broken linking might have kept it from running correctly.  I don't have any other explanation.

It turns out this was not (best I can tell) related to the new year time sync issues that wer seen at the sites.

RTS/RCG/DAQ UPGRADE TO COMMENCE

I will be attempting (again) to upgrade the RTS, including the RCG and the daqd, to version 2.4 today.  The RTS will be offline until further notice.

After running into more problems with the upgrade, I eventually decided to abort todays upgrade attempt, and revert back to where we were this morning (RTS 2.1).  I'll try to follow this with a fuller report explaining what problems I encountered when attempting the upgrade.

However, when Alex and I were trying to figure out what was going wrong in the upgrade, it appears that the fb symmetricom card lost the ability to sync with the GPS receiver.  When the symmeticom module is loaded, dmesg shows the following:

[  285.591880] Symmetricom GPS card on bus 6; device 0
[  285.591887] PIC BASE 2 address = fc1ff800
[  285.591924] Remapped 0x17e2800
[  285.591932] Current time 947125171s 94264us 800ns 
[  285.591940] Current time 947125171s 94272us 600ns 
[  285.591947] Current time 947125171s 94280us 200ns 
[  285.591955] Current time 947125171s 94287us 700ns 
[  285.591963] Current time 947125171s 94295us 800ns 
[  285.591970] Current time 947125171s 94303us 300ns 
[  285.591978] Current time 947125171s 94310us 800ns 
[  285.591985] Current time 947125171s 94318us 300ns 
[  285.591993] Current time 947125171s 94325us 800ns 
[  285.592001] Current time 947125171s 94333us 900ns 
[  285.592005] Flywheeling, unlocked...

Because of this, the daqd doesn't get the proper timing signal, and consequently is out of sync with the timing from the models.

It's completely unclear what caused this to happen.  The card seemed to be working all day today, then Alex and I were trying to debug some other(maybe?) timing issues and the symmetricom card all of a sudden stopped syncing to the GPS.  We tried rebooting the frame builder and even tried pulling all the power to the machine, but it never came back up.  We checked the GPS signal itself and to the extend that we know what that signal is supposed to look like it looked ok.

I speculate that this is also the cause of the problems were were seeing earlier in the week.  Maybe the symmetricom card has just been acting flaky, and something we did pushed it over the edge.

Anyway, we will try to replace it tomorrow, but Alex is skeptical that we have a replacement of this same card.  There may be a newer Spectracom card we can use, but there may be problems using it on the old sun hardware that the fb is currently running on.  We'll see.

In the mean time, the daqd is running rogue, off of it's own timing.  Surprisingly all of the models are currently showing 0x0 status, which means no problems.  It doesn't seem to be recording any data, though.  Hopefully we'll get it all sorted out tomorrow.

Alex and I have taken the framebuilder offline to try to see what's wrong with the symmetricom card.  We have removed the card from the chassis and Alex has taken it back to downs to do some more debugging.

We have been formulating some alternate methods to get timing to the fb in case we can't end up getting the card working.

The framebuilder is back online now, minus it's symmetricom GPS card.  The card seems to have failed entirely, and was not able to be made to work at downs either.  It has been entirely removed from fb.

As a fall back, the system has been made to work off of the system NTP-based time synchronization.  The latest symmetricom driver, which is part of the RCG 2.4 branch, will fall back to using local time if the GPS synchronization fails.  The new driver was compiled from our local checkout of the 2.4 source in the new to-be-used-in-the-future rtscore directory:

controls@fb ~ 0$ diff {/opt/rtcds/rtscore/branches/branch-2.4/src/drv/symmetricom,/lib/modules/2.6.34.1/kernel/drivers/symmetricom}/symmetricom.ko
controls@fb ~ 0$ 

The driver was reloaded.   daqd was also linked against the last running stable version and restarted:

controls@fb ~ 0$ ls -al $(readlink -f /opt/rtcds/caltech/c1/target/fb/daqd)
-rwxr-xr-x 1 controls controls 6592694 Dec 15 21:09 /opt/rtcds/caltech/c1/target/fb/daqd.20120104
controls@fb ~ 0$ 

We'll have to keep an eye on the system, to see that it continues to record data properly, and that the fb and the front-ends remain in sync.

The question now is what do we do moving forward.  CDS is not supporting the symmetricom cards anymore, and have moved to using Spectracom GPS/IRIG-B cards.  However, Downs has neither at the moment.  Even if we get a new Spectracom card, it might not work in this older Sun hardware, in which case we might need to consider upgrading the framebuilder to a new machine (one supported by CDS).

While Kiwamu and I were trying to investigate the the vertex glitches we were noticing excess noise in ITMX, which Kiwamu blamed on some sort of bad diagonalization.  Sure enough, the ITMX input matrix is in the default state [0], not a properly diagonalized state.  Looking through the rest of the suspensions, I found PRM also in the default state, not diagonalized.

We should do another round of suspension diagonalization.

Kiwamu (or whoever is here last tonight): please run the free-swing/kick script (/opt/rtcds/caltech/c1/scripts/SUS/freeswing) before you leave, and I'll check the matrices and update the suspensions tomorrow morning.

[0]

I retrieved the newly "secured" router from Junaid.  It had apparently been hooked up to the GC network via it's LAN port, but it's LAN services had no been shut off.  It was therefore offering a competing DHCP server, which will totally hose a network.  A definite NONO.

The new SSID is "40mWiFi", it's WPA2, and the password is pasted to the bottom of the unit (the unit is back in it's original spot on the office computer rack.

The RAID (JetStor SATA 416S) is indeed resyncing itself after a disk failure.  There is a hot spare, so it's stable for the moment.  But we need a replacement disk:

    RAID disks:  1000.2GB Hitachi HDT721010SLA360

Do we have spares?  If not we should probably buy some, if we can.  We want to try to keep a stock of the same model number.

Other notes:

The RAID has a web interface, but it was for some reason not connected.  I connected it to the martian network at 192.168.113.119.

Viewing the RAID event log on the web interface silences the alarm.

I retrieved the manual from Alex, and placed it in the COMPUTER MANUALS drawer in the filing cabinet.

We reset the interlock and restarted the PSL.  The end AUX lasers seem to have come back online fine.  PMC and mode cleaner locked back up quickly.

I am continuing to attempt to upgrade the CDS system to RTS 2.5.  Systems will continue to be up and down for the rest of the day.

Upgrade progresses, but not complete.  There are some relatively minor issues, and one potentially big issue.

All new software has been installed, including the new epics that supports long channel names.

I've been doing a LOT of cleanup.  It was REALLY messy in there.

The new framebuilder/daqd code is running on fb.

Models are compiling with the new RCG and I am able to get them running.  Some of them are not compiling for relatively minor reasons (the simulink models need updating).  I'm also running into compile problems with IOPs that are using the dolphin drivers.

The major issue is that the framebuilder and the models are not syncing their timing, so there's no data collection.  I've spoken to Alex and he and Rolf are going to come over tomorrow to sort it out.  It's possible that we're missing timing hardware that the new code is expecting.

There are still some stability issues I haven't sorted out yet, and I have a lot more cleanup to do.

At this rate I'm going to shoot for being done Thursday.

Apparently there was a catastrophic power failure last night.  Bob says it took out power in most of Pasadena.

Bob checked the vacuum system when he got in first thing this morning and everything's back up and checks out.  The laster is still off and most of the front-end computers did not recover.

I'm going to start a boot fest now.  I'll be able to report more once everything is back on.

All of the front-ends are back up and I've been able to recover local control of all of the opitcs (snapshots from saturday). Issues:

• I can't lock the PMC.  Still unclear why.
• there are no oplev signals in MC1, MC2, and MC3
• Something is wrong with PRM.  He is very noisy.  Turning on his oplev servo makes him go crazy.
• There are all sorts of problems with the oplevs in general.  Many of the optics have no oplev settings.  This is probably not related to the power outage.

On a brighter note, ETMX is damped with it's new RCG 2.5 controller!  yay!

The upgrade is nearly complete:

• new daqd code is running on fb
• the fe/daqd timing issue was resolved by adjusting the GPS offset in the daqdrc.  I will document this more later.
• the power outage conveniently rebooted all the front-end machines, so they're all now running new caRepeater
• all models have been successfully recompiled with RCG 2.5 (with only a couple small glitches)
• all new models are running on all front-end machines (with a couple exceptions)
• all suspension models seem to be damping under local control (PRM is having troubles that are likely unrelated to the upgrade).
• a lot of cleanup has been done

Remaining tasks/issues:

• more testing OF EVERYTHING needs o be done
• I did not yet update the DIS dolphin code, so we're running with the old code.  I don't think this is a problem, but it would be nice to get us running what they're running at the sites
• I tried to cleanup/simplify how front-end initialization is done.  However, there is a problem and models are not auto-starting after reboot.  This needs to be fixed.
• the userapps directory is in a new place (/opt/rtcds/userapps).  Not everything in the old location was checked into the repository, so we need to check to make sure everything that needs to be is checked in, and that all the models are running the right code.
• the c1oaf model seems to be having a dolphin issue that needs to be sorted
• the c1gfd model causes c1ioo to crash immediately upon being loaded.  I have removed it from the rtsystab.  That model needs to be fixed.
• general model cleanup is in order.
• more front-end cleanup is needed, particularly in regards to boot-up procedure.
• document the entire upgrade procedure.

I'll finish up these remaining tasks tomorrow.

 Yes, things have changed.  Please wait until I have things cleaned up before working on models.  I'll explain what the new setup is.

I ran into a couple of snags today.

A big one is that the framebuilder daqd started going haywire when I told it to start writing frames.  After restart the logs started showing this:

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

and the network seemed like it started to get really slow.  I wasn't able to figure out what was going on, so I shut the frame writing off again.  I'll have to work with Rolf on that next week.

Another big problem is the workstation application upgrades.  The NDS protocol version has been incremented, which means that all the NDS client applications have to be upgraded.  The new dataviewer is working fine (on pianosa), but dtt is not:

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$ 

I don't know what's going on here.  All the library paths are ok.  Hopefully I'll be able to figure this out soon.  The old version of dtt definitely does not work with the new setup.

I might go ahead and upgrade some more of the workstations to Ubuntu in the next couple of days as well, so everything is more on the same page.

I also tried to cleanup the front-end boot process, which has it's own problems (models won't auto-start).  I haven't figured that out yet either.  It really needs to just be completely overhauled.

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$

 dtt (diaggui) and dataviewer are now working on pianosa to retrieve realtime data and past data from DQ channels.

Unfortunately it looks like there may be a problem with trend data, though.  If I try to retrieve 1 minute of "full data" with dataviewer for channel C1:SUS-ITMX_SUSPOS_IN1_DQ around GPS 1019089138 everything works fine:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

but if I specify any trend data (second, minute, etc.) I get the following:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex warned me that this might have happened when I was trying to test the new daqd without first turning off frame writing.

I'm not sure how to check the integrity of the frames, though.  Hopefully they can help sort this out on Monday.

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

So the framebuilder/daqd frame writing issue seems to have been a transient one.  Alex said he tried enabling frame writing manually and it worked fine, so I tried re-enabling the frame writing config lines and sure enough it worked fine.  So it's a mystery.  Alex said the "main profiler warning" lines tend to show up when data is backed up in the buffer, although he didn't explain why exactly that would have been the issue here.

daqdrc frame writing directives:

start frame-saver;
sync frame-saver;
start trender;
start trend-frame-saver;
sync trend-frame-saver;
start minute-trend-frame-saver;
sync minute-trend-frame-saver;
start raw_minute_trend_saver;

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex told me that the "trend data is not available" message comes from the "trender" functionality not being enabled in daqd.  After re-enabling it (see #6555) minute trend data was available again.  However, there still seems to be an issue with second trends.  When I try to retrieve second trend data from dataviewer for which minute trend data *is* available I get the following error message:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

Awaiting more help from Alex...

The new rtcds utility

I have written a new utility for interacting with the CDS RTS/RCG system: "rtcds".  It should be available on all workstations and front-end machines, but certain commands are restricted to run on certain front end machines (build, start, stop, etc.).  Here's the help:

controls@c1lsc ~ 0$ rtcds help
Usage: rtcds <command> [arg]

Available commands:

  build|make SYS      build model
  install SYS         install model
  start SYS|all       start model
  restart SYS|all     restart running model
  stop|kill SYS|all   stop running model
  list                list all models for current host

controls@c1lsc ~ 0$ 

Please use this new utility from now on when interacting with RTS.

I'll be improving and expanding it as we go.   Please let me know if you encounter any problems.

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

It looks like this is actually just a limit of how long we're saving the second trends, which is just not that long.  I'll look into extending the second trend look-back.

NOTE: Unless you really care about what's going on under the hood, please ignore this entire post and go here: USE THE NEW RTCDS UTILITY

 An important new aspect of this upgrade is that we have some new directories and some of the code paths have moved to correspond with the "standard" LIGO CDS filesystem hierarchy:

• rtscore:      /opt/rtcds/rtscore/release       -->  RTS RCG source code (svn: adcLigoRTS)
• userapps: /opt/rtcds/userapps/release  -->  CDS userapps source for models, RCG C code, medm screens, scripts, etc (svn: cds_user_apps)
• rtbuild:       /opt/rtcds/caltech/c1/rtbuild    -->  RCG build directory

All work should be done in the "userapps" directory, and all builds should be done in the build directory.  Some important points:

WARNING: DO NOT MODIFY ANYTHING IN RTSCORE

This is important.  The rtscore directory is now just where the RCG source code is stored.  WE NO LONGER BUILD MODELS IN THE RTS SOURCE  Please use the rtbuild directory instead.

NO MORE MODEL/CODE SYMLINKS

You don't need to link you model or code anywhere to get it to compile.  The RCG now uses proper search paths to source in the RCG_LIB_PATH to find the needed source.  This has been configured by the admin, so as long as you put your code in the right place it should be found.

ALL CODE/MODELS/ETC GO IN USERAPPS

All RTS code is now stored ENTIRELY in the userapps directory (e.g. /opt/rtcds/userapps/release/isc/c1/models/c1lsc.mdl).  This is more-or-less the same as before, except that symlinking is no longer necessary.  I have placed a symlink at the old location for convenience.

BUILD ALL MODELS IN RTSCORE

You can run "make c1lsc" in the rtbuild directory, just as you used to in the old core directory.  However, don't do that.  USE THE NEW RTCDS UTILITY instead.

This morning I installed Ubuntu 10.04 on rosalba.  This is the same version that is running on pianosa.  The two machines should be identically configured, although rosalba may be missing some apt-getable packages.