40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 78 of 339  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  16478   Mon Nov 22 16:38:26 2021 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Tega, Ian]


1. Investigate cross-coupling btw the various degrees of freedom (dof) - turn on noise for each dof in the plant model and measure the transfer function of the other dofs.

2. Get a closed-loop transfer function using noise injection and give a detailed outline of the procedure in elog - IN1/IN2 for each TM_RESP filter while the others are turned off.

3. Derive analytic model of the closed-loop transfer functions for comparison.

4. Adapt control filters to fit optimized analytical solutions.

  16615   Mon Jan 24 17:10:25 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Ian, Tega]

Connected the New SUS screens to the controller for the simplant model. Because of hard-coded links in the medm screen links, it was necessary to create the following path in the c1sim computer, where the new medm screen files are located:



We noticed a few problems:

1. Some of the medm files still had C1 hard coded, so we need to replace them with $IFO instead, in order for the custom damping filter screen to be useful.

2. The "Load coefficient" button was initially blank on the new sus screen, but we were able to figure out that the problem came from setting the top-level DCU_ID to 63.




Get the data showing the controller damping the pendulum. This will involve tweaking some gains and such to fine-tune the settings in the controller medm screen. Then we will be able to post some data of the working controller.


[Useful aside]

We should have a single place with all the instructions that are currently spread over multiple elogs so that we can better navigate the simplant computer.

Attachment 1: Screen_Shot_2022-01-24_at_5.33.15_PM.png
  16626   Thu Jan 27 16:40:57 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Ian, Paco, Tega]

Last night we set up the four main matrices that handle the conversion between the degrees of freedom bases and the sensor bases. We also wrote a bash script to automatically set up the system. The script sets the four change of bases matrices and activates the filters that control the plant. this script should fully set up the plant to its most basic form. The script also turns off all of the built-in noise generators. 

After this, we tried damping the optic. The easiest part of the system to damp is the side or y motion of the optic because it is separate from the other degrees of freedom in both of the bases. We were able to damp that easily. in attachment 1 you can see that the last graph in the ndscope screen the side motion of the optic is damped. Today we decided to revisit the problem.

Anyways, looking at the problem with fresh eyes today, I noticed the in pit2pit coupling has the largest swing of all the plant filters and thought this might be the reason why the inputs (UL,UR,LR,LL) to the controller was hitting the rails for pit DoF. I reduce the gain of the pit2pit filter then slowly increased it back to one. I also reduced the gain in the OSEM input filter from 1 to 1/100. The attached image (Attachment2) is the output from this trial. This did not solve the problem. The output when all OSEM input filter gain set to one is shown in Attachment2.

We will try to continue to tweak the coefficients. We are probably going to ask Anchal and Paco to sit down with us and really hone in on the right coefficients. They have more experience and should be able to really get the right values.

Attachment 1: simplant_control_1.png
Attachment 2: simplant_control_0.png
  16645   Thu Feb 3 17:15:23 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

Finally got the SIMPLANT damping to work following Rana's suggestion to try damping one DoF at a time, woo-hoo!

At first, things didn't look good even when we only focus on the POS DoF. I then noticed that the input value (X1:SUS-OPT_PLANT_TM_RESP_1_1_IN1) to the plant was always zero. This was odd bcos it meant the control signal was not making its way to the plant. So I decided to look at the sensor data


that adds up via the C2DOF matrix to give the POS DoF and I noticed that these interior nodes can take on large values but always sum up to zero because the pair (UL, LL) was always the negative of (UR,LR). These things should have the same sign, at least in our case where only the POS DoF is excited, so I tracked the issue back to the alternating (-,+,-,+,-) convention for the gains


of the Coil Output filters used in the real system, which we adopted in the hopes that all was well. Anyways, I changed them all back to +1. This also means that we need to change the sign of the gain for the SIDE filter, which I have done also (and check that it damps OK). I decided to reduce the magnitude of the SIDE damping from 1 to 0.1 so that we can see the residuals since the value of -1 quickly sends the error to zero.  I also increased the gain magnitude for the other DoF to 4.

When looking at the plot remember that the values actually represent counts with a scaling of 2^15 (or 32768) from the ADC. I switched back to the original filters on FM1 (e.g. pit_pit ) without damping coefficients present in the FM2 filter (e.g. pit_pit_damp).

FYI, Rana used the ETMY suspension MEDM screen to illustrate the working of the single suspension to me and changed maybe POS and PITCH gains while doing so.

Also, the Medify purifier 'replace filter' indicator issue occurred bcos the moonlight button should have been pressed for 3 seconds to reset the 'replace filter' indicator after filter replacement.

Attachment 1: Screen_Shot_2022-02-03_at_8.23.07_PM.png
  16654   Wed Feb 9 14:34:27 2022 IanSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

Restarted the C1sim machine at about 12:30 to help diagnose a network problem. Everything is back up and running

Attachment 1: SummaryMdemScreen.png
  17036   Tue Jul 26 19:50:25 2022 DeekshaUpdateComputer Scripts / ProgramsVector fitting

Trying to vectfit to the data taken from the DFD previously but failing horribly. I will update this post as soon as I get anything semi-decent. For now here is this fit.

Attachment 1: data.png
Attachment 2: fit_attempt.png
  17038   Tue Jul 26 21:16:41 2022 KojiUpdateComputer Scripts / ProgramsVector fitting

I think the fit fails as the measurement quality is not good enough.


  17075   Thu Aug 11 16:48:59 2022 ranaUpdateComputer Scripts / ProgramsNDS2 updates

We had several problems with our NDS2 server configuration. It runs on megatron, but I think it may have had issues since perhaps not everyone was aware of it running there.

  1. channel lists were supposed to updated regularly, but the nds2_nightly script did not exist in the specified directory. I have moved it from Joe Areeda's personal directory (/home/nds2mgr/joework/server/src/utils/) to nds2mgr/channel-tracker/.
  2. The channel history files (/home/nds2mgr/channel-tracker/channel_history/) are stored on the local megatron disk. These files had grown up to ~50 GB over tha past several years. I backed these up to /users/rana/, and then wiped them out so that the NDS could regen them. Now that the megatron local disk is not full, it seems to work in giving raw data.
  3. Need to confirm that this serves up trend data (second and minute)
  4. I think there is a nds2-server package for Debian, so we should update megatrons OS to the preferred flavour of DebIan and use that. Who to get to help in this install?

Since Megatron is currently running the "Shanghai" Quad-core Opteron processor from ~2009,  its ~time to replace it with a more up to date thing. I'll check with Neo to see if he has any old LDAS leftovers that are better.

  32   Tue Oct 30 19:32:13 2007 tobinProblem FixedComputersconlogger restarted
I noticed that the conlogger wasn't running. It looks like it hasn't been running since October 11th. I modified the restart_conlogger script to insist that it run on op340m instead of op440m, and then ran it on op340m.
  46   Thu Nov 1 16:34:47 2007 Andrey RodionovSummaryComputersLimitation on attachment size of E-LOG

I discovered yesterday when I was attaching photos that it is NOT possible to attach files whose size is 10Mb or more. Therefore, 10Mb or something very close to that value is the limit.
  71   Tue Nov 6 16:48:54 2007 tobinConfigurationComputersscopes on the net
I configured our two 100 MHz Tektronix 3014B scopes with IP addresses: (scope0) and (scope1). Let the scripting commence!

There appears to be a Matlab Instrument Control Toolbox driver for this scope.
  72   Tue Nov 6 18:18:15 2007 tobinConfigurationComputersI broke (and fixed) conlogger
It turns out that not only restart_conlogger, but also conlogger itself checks to see that it is running on the right machine. I had changed the restart_conlogger script to run on op340, but it would actually silently fail (because we cleverly redirect conlogger's output to /dev/null). Anyway, it's fixed now: I edited the conlogger source code where the hostname is hardcoded (blech!) and recompiled.

On another note, Andrey fixed the "su" command on op440m. It turns out that the GNU version, in /usr/local/bin, doesn't work, and was masking the (working) sun version in /bin. Andrey renamed the offending version as "su.backup".
  73   Tue Nov 6 23:45:38 2007 tobinConfigurationComputerstektronix scripts!
I cooked up a little script to fetch the data from the networked Tektronix scope. Example usage:

linux2:scripts>tektronix/tek-dump scope0 ch1 foo.csv

"scope0" is the hostname of the scope, "ch1" is the channel you want to dump, and "foo.csv" is the file you want to dump it to. The script is written in Python since Python's libhttp gave me less trouble than Perl's HTTP::Lite.
  77   Wed Nov 7 10:55:21 2007 ajwConfigurationComputersbackup script restarted
Following the reboot of computers on 10/31/07, the backup script required restart (which unfortunately "can't" be automated because a password needs to be typed in). I restarted, following the instructions in /cvs/cds/caltech/scripts/backup/000README.txt and verified that it more-or-less worked last night (the rsync sometimes times out; it gets through after a couple of days of trying.)
  92   Sun Nov 11 21:21:04 2007 ranaHowToComputersNew DV
To use the new ligoDV (previously GEO DV) to look at 40m data, open up a matlab, set up for mDV as usual,
and then from the /cvs/cds/caltech/apps/ligoDV/ directory, type 'ligoDV'.

Then select which NDS server you want to look at and then start clicking to get some plots.
Attachment 1: Screenshot-1.png
  106   Thu Nov 15 18:06:06 2007 tobinUpdateComputersalex: linux1 root file system hard disk's dying
I just noticed that Alex made an entry in the old ilog yesterday, saying: "Looks like linux1 root filesystem hard drive is about to die. The system log is full of drive seek errors. We should get a replacement IDE drive as soon as possible or else the unthinkable could happen. 40 Gb IDE hard drive will be sufficient."
  107   Thu Nov 15 18:23:55 2007 JohnHowToComputersSwap CAPS and CTRL on a Windows 2000/XP machine
I've swapped ctrl and caps on the four control room Windows machines. Right ctrl is unchanged.

Start menu->Run "regedit"

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout

Click on the KeyboardLayout entry.

Edit->New Binary Value name it Scancode Map.

Then select the new Scancode Map entry.

Edit menu->Modify Binary Data.

In the dialog box enter the following data:

0000: 00 00 00 00 00 00 00 00
0008: 03 00 00 00 3A 00 1D 00
0010: 1D 00 3A 00 00 00 00 00

Exit the Registry Editor. You need to log off and then on in XP (and restart in Windows 2000) for the changes to be made.
  109   Thu Nov 15 18:37:06 2007 tobinUpdateComputerspossible replacement for linux1's disk
It looks like the existing disk in linux1 is a Seagate ST380013A (this can be found either via the smartctl utility or by looking at the file /proc/ide/hda/model). It appears that you can still buy this disk from amazon, though I think just about any ATA disk would work. I'll ask Steve to buy one for us.
  110   Fri Nov 16 11:27:18 2007 tobinUpdateComputersscript fix
I added a tidbit of code to "LIGOio.pm" that fixes a problem with ezcastep on Linux. Scripts such as "trianglewave" will now work on Linux.
# On Linux, "ezcastep" will interpret negative steps as command line arguments,
# because the GNU library interprets anything starting with a dash as a flag.
# There are two ways around this.  One is to set the environment variable
# POSIXLY_CORRECT and the other is to inject "--" as a command line argument
# before any dashed arguments you don't want interpreted as a flag.  The former
# is easiest to use here:

if (`uname` =~ m/Linux/) {
    # Add an environment variable for child processes
  111   Fri Nov 16 14:11:26 2007 tobinUpdateComputersop140
Alan called to say that Phil Ehrens will be coming by to take op140 off our hands.
  112   Fri Nov 16 14:31:43 2007 tobinUpdateComputersop140 disks
Phil Ehrens stopped by and took op140's disks.
Attachment 1: DSC_0173.JPG
  117   Tue Nov 20 11:10:07 2007 tobinUpdateComputersepics access from matlab
I installed "labca", which allows direct access to EPICS channels from within Matlab. It comes with both Linux and Solaris binaries (and source) but I've only tried it on linux.

To set it up, run these shell commands:
pushd /cvs/cds/caltech/users/tf/build/labca_2_1/bin/linux-x86
setenv PATH ${PATH}:`pwd`
cd /cvs/cds/caltech/users/tf/build/labca_2_1/lib/linux-x86
Then start matlab, and within matlab type:
addpath /cvs/cds/caltech/users/tf/build/labca_2_1/bin/linux-x86/labca
help labca
foo = lcaGet('C1:PSL-FSS_RCTRANSPD')
It seems like reasonably well-written software, and is being actively maintained right now. If we like it, I can build a more recent version, install it in a more permanent location, etc.
  118   Tue Nov 20 13:06:57 2007 tobinConfigurationComputerslinux1 has new disk
Alex put the new hard disk into linux1 along with a fresh install of linux (CentOS). The old disk was too damaged to copy.

Alex speculates that the old disk failed due to overheating and that linux1 could use an extra fan to prevent this in the future.
  119   Tue Nov 20 18:02:54 2007 JohnSummaryComputersPSL_Main screen
I've updated the PSL_MAIN screen. The old version may be found in cvs/cds/caltech/medm/old/medm/psl.
Attachment 1: PSL_Screen.tif
  120   Tue Nov 20 18:35:20 2007 JohnHowToComputersMatLab in Emacs
If you can't get MatLab to run in emacs try adding the following to the .emacs file

(setq matlab-shell-command-switches '("-nojvm"))

This stops the gui opening.

To start MatLab type M-x matlab-shell.
To enter MatLab mode M-x matlab-mode.

I've done this on LINUX3.

To run MatLab in emacs under windows one can use MatLabShell http://www.cs.umb.edu/~ram/matlabShell/index.html
  132   Wed Nov 28 16:46:28 2007 ranaConfigurationComputersscientific linux 5.0
I tried installing Scientific Linux on Tiramisu. The installation process was so bad (really)
that I quit after 15 minutes. Its back to booting Ubuntu as if nothing had ever happened. Let
us never speak of Scientific Linux again.
  140   Thu Nov 29 14:29:22 2007 tobinConfigurationComputerslinux1 httpd/conlogger fixed
I think I fixed the conlogger web interface on linux1.

Steps necessary to do this:
0. Run "/etc/init.d/httpd start" to start up httpd right now
1. Run "/usr/sbin/ntsysv" and configure httpd to be started automatically in the future
2. Copy /cvs/cds/caltech/conlogger/bin/conlog_web.pl to /var/www/cgi-bin and chown to controls
8. Hack the conlog_web.pl to (0) use /usr/bin/perl (1) not use Apache::Util, and (2) function with the newer version of CGI.pm
9. Enjoy!

The following steps are optional, and may be inserted between steps 2 and 8:
3. Try to install Apache::Util (via "perl -MCPAN -e shell" followed by "Install Apache::Util")
4. Notice that the installation dies because there is no C compiler installed
5. Bang head in disgust and abomination over a Linux distribution shipping without a C compiler installed by default
6. "yum install gcc"
7. Annoyed by further dependencies, go to step 8
  149   Fri Nov 30 19:46:58 2007 ranaConfigurationComputersEPICS Time Bad again
The time on the EPICS screens is off by 10 minutes again. Por Que?

Its because the ntpd on scipe25 wasn't restarted after the last boot. If someone
knows how to put the ntpd startup into that machine, please do so.

This time I started it up by just going sshing in as controls and then entering:

sudo /usr/sbin/ntpd -c /etc/ntp.conf

which runs it as root and points to the right file.

It takes a few minutes to get going because all of the martian machines have to first fail to
connect to the worldwide pool servers (e.g. 0.pool.ntp.org) before they move on and try linux1
which has a connection to the world. Once it gets it you'll see the time on the EPICS screens
freeze. It then waits until the ntp time catches up with its old, wrong time before updating

According to the Wikipedia, this time is then good to 128 ms or less.
  153   Sun Dec 2 17:37:33 2007 ranaOmnistructureComputersNetwork Cabling in the Office
We all know that we've spent many integrated man hours trying to figure out why our network connections
in the office area don't work. Usually its because of the bad hub around the Tobin/Osamu desk.

I pried open some of the wall conduit today and it looks pretty easy to fish cables through. I think
its time we finally did that. It may be a little disruptive, but I propose we get Larry to come over
and figure out what needs to happen for us to get regular 100 Mbit ports on the walls. These can
then all go over and get connected to a switch in the rack that holds linux1.

Opinions / comments ?
  158   Mon Dec 3 16:24:47 2007 tobinHowToComputersGNU screen
GNU screen is a utility that can be quite handy for managing long-running psuedo-interactive terminal programs on remote machines. In particular, I think it might be useful in developing and testing "Matlab DMT" tools on Mafalda.
  164   Wed Dec 5 10:57:08 2007 albertoHowToComputersConnecting the GPIBto USB interface to the Dell laptop
The interface works only on one of the USB ports of the laptop (the one on the right, looking at the computer from the back).
  192   Sun Dec 16 16:52:40 2007 dmassUpdateComputersComputer on the end Fixed
I had Mike Pedraza look at the computer on the end (tag c21256). It was running funny, and turns out it was a bad HD.

I backed up the SURF files as attachments to their wiki entries. Nothing else seemed important so the drive was (presumably) swapped, and a clean copy of xp pro was installed. The username/login is the standard one.

Also - that small corner of desk space is now clean, and it would be lovely if it stayed that way.
  217   Thu Dec 27 18:18:56 2007 ranaUpdateComputersUpdate on GigE Camera

So I finally got the linux software to compile on mafalda. I got the software to dump all the information regarding the camera onto a file. I tried to take a tiff snap and came up empty. So I looked at the configuration file and realized that the camera thinks that the frame-rate is a nan. Am reading up the manual to fix the frame-rate manually and then will attempt to take another snap.

All the files are in a folder called Prosilica in /home/controls/ on mafalda. All the executables are in /home/controls/Prosilica/bin-pc/x86/* . On another note, I am looking for a name for the camera. Any suggestions are welcome.

Suggestion #1: put this in the target area in a directory called /prosilica/. /home/controls is not backed up.

Suggestion #2: put a readme file in there on any work that was necessary to get it to compile.

Suggestion #3: make a wiki page for the camera with all the info that camera code developers will need
  228   Wed Jan 9 10:47:15 2008 frickeUpdateComputersethernet wiring in office/control room
The electrical people have been here for three days, installing ethernet cables and drops in the 40m office area, in the old conduits where there was power and 10base2. Soon we will have reliable ethernet connections, instead of relying on switches hanging from the ceiling, etc!
  274   Sat Jan 26 18:12:45 2008 ranaHowToComputersextra processes and crashes
There were a load of extra processes on op440m; they were all related to rogue DV processes (frameMemRead, framer4, xmgrace, etc.)

You can check this by running top. IF there's lots of them you can kill them by using 'pkill'.

I'm attaching the screen cap of a 'top' session. You can also see that the cron of updatedb is running and taking upp all the CCPU. Thee result is that the delay puts misspellings and doouble characters intoo my elog entry. Clearly wwe'll have to fix the cron setup.
  275   Sat Jan 26 18:48:43 2008 JohnSummaryComputersRestart iscepics
iscepics died this afternoon. We restarted it and restored settings from yesterday. I've written up instructions in the wiki.
  281   Mon Jan 28 17:16:54 2008 AndreyConfigurationComputersMatlab libraries DO NOT WORK properly sometimes

Working in Matlab, I encountered at two different times today the license distribution problem:

??? License checkout failed.
License Manager Error -4
Maximum number of users for Curve_Fitting_Toolbox reached.
Try again later.
To see a list of current users use the lmstat utility or contact your License Administrator.

Troubleshoot this issue by visiting:
  294   Sat Feb 2 14:11:27 2008 JohnSummaryComputersOPTLEVmaster screen
I changed the layout of the optlev master screen. The old version is /cvs/cds/caltech/medm/old/C1ASC_OPTLEVmaster080202.adl
  302   Fri Feb 8 17:09:52 2008 Max JonesUpdateComputersChanges to NoiseBudget
Today I altered the following files

In DARM_CTRL case I changed the second channel name to DARM_ERR. Messy but it may be effective.

I commented out lines of code specifically pertaining to non-existent DARM_DAQ channel. Marked in
code around approximately line 60.

Please address all comments or concerns to me (williamj(at symbl)caltech.edu). Thank you.
  318   Thu Feb 14 17:21:53 2008 Max JonesUpdateComputersNoise budget code changes
In cvs/cds/caltech/NB/matlab/utilities/LSCmodel.m at line 146
I have hardwired in changes to struct lsc. Please see code.
  319   Fri Feb 15 10:28:44 2008 ranaUpdateComputersNoise budget code changes

In cvs/cds/caltech/NB/matlab/utilities/LSCmodel.m at line 146
I have hardwired in changes to struct lsc. Please see code.

The IFOin variable (which I admit is not documented) should refer to a file called
C1IFOparams.m in the ReferenceData directory. This does not yet exist but can be
created by merging L1IFOparams.m with our knowledge of the 40m IFO.
  320   Fri Feb 15 22:16:04 2008 AndreyUpdateComputersMATLAB is not working: "Licence checkout failed"

For some unknown to me reason,

Matlab stopped working about 20 minutes ago on all computers in the control room (both UNIX control machines and Windows).

It says: "License checkout failed. License Manager Error -15. MATLAB is unable to connect to the license server."

I do not know how to revive Matlab.

At the same time, I consider that I made a significant progress in building my theoretical/computational model in the last 2 days. I was able to compute the time-evolution of accelerometer signals through stacks and pendulums using Matlab command "lsim", and I am now able to calculate RMS of spectrum of differential arm length in different frequency intervals. It seemed to me that everything is ready in my program to make the three-dimensional theoretical/computational plot (RMS as a function of Q-factors of ETMX and ITMX), but unfortunately Matlab stopped working. It seemed to me that all that was remaining was to run a loop with all possible values of Q-factors. Let's hope that Matlab will be working after the weekend.

  345   Thu Feb 28 16:19:37 2008 josephbConfigurationComputersMafalda rewired and multiple cameras running
1) Mafalda is now connected via an orange Cat5E ethernet cord to the gigabit ethernet switch in rack in the office space. It has been labeled at both ends with "mafalda".

2) Both the GC650M camera (from MIT) and the GC750M are working. I can run the sampleviewer code and get images simultaneously. Unforutnately, the fps on both cameras seems to drop roughly in half (not an exact measurement) when displaying both simultaneously at full resolution.

3)Discovered the Gigabit ethernet card in Mafalda doesn't support jumbo packets (packets of up to 9k bytes), which is what they recommend for optimum speed.

4)However, connecting the cameras through only gigabit switches to Mafalda did seem to increase the data rate anyways, roughly by a factor of 2. (Used to take about 80 seconds to get 1000 frames saved, now takes roughly 40 seconds).

5)Need to determine the bottleneck on the cameras. It may be the ethernet card, although its possible to connect multiple gigabit cards to single computer (depending on the number of PCI slots it has). Given the ethernet cards are cheap ($300 for 20) compared to even a single camera (~$800-1500), it might be worth while outfitting a computer with multiple.
  346   Thu Feb 28 19:37:41 2008 robConfigurationComputersmultiple cameras running and seisBLRMS

1) Mafalda is now connected via an orange Cat5E ethernet cord to the gigabit ethernet switch in rack in the office space. It has been labeled at both ends with "mafalda".

2) Both the GC650M camera (from MIT) and the GC750M are working. I can run the sampleviewer code and get images simultaneously. Unforutnately, the fps on both cameras seems to drop roughly in half (not an exact measurement) when displaying both simultaneously at full resolution.

3)Discovered the Gigabit ethernet card in Mafalda doesn't support jumbo packets (packets of up to 9k bytes), which is what they recommend for optimum speed.

4)However, connecting the cameras through only gigabit switches to Mafalda did seem to increase the data rate anyways, roughly by a factor of 2. (Used to take about 80 seconds to get 1000 frames saved, now takes roughly 40 seconds).

5)Need to determine the bottleneck on the cameras. It may be the ethernet card, although its possible to connect multiple gigabit cards to single computer (depending on the number of PCI slots it has). Given the ethernet cards are cheap ($300 for 20) compared to even a single camera (~$800-1500), it might be worth while outfitting a computer with multiple.

I found the SampleViewer running and displaying images from the two cameras. This kept mafalda's network so busy that the seisBLRMS program fell behind by a half-hour from its nominal delay (so 45 minutes instead of 12), and was probably getting steadily further behind. I killed the SampleViewer display on linux2, and seisBLRMS is catching up.
  352   Mon Mar 3 13:58:10 2008 steveUpdateComputers RFM Network are down
The CODAQ_RFNETWORK are down, except C1SUSVME & AWG
  353   Mon Mar 3 19:34:40 2008 ranaUpdateComputers RFM Network are down

The CODAQ_RFNETWORK are down, except C1SUSVME & AWG

All of the FE machines were found to be down this afternoon. I called Alex and he suggested several
things which didn't work (restart EPICS tasks, power cycle RFM switch, etc.).

Then he suggested that I go around and power cycle every crate!!! And that sometimes the order of this
matters!!! I think he was just recording this conversation so that he could have a laugh by playing it on Youtube.

However, power cycling all of the FE crates seemed to work. Alex's theory is that 'something goes bad in the
RFM cards sometimes'.

Its all green now.
  354   Tue Mar 4 00:42:51 2008 ranaUpdateComputersFB0 still down ?
The framebuilder is still down. I tried restarting the daqd task and resetting the RFM
switch like it says in the Wiki but it still doesn't work right. The computer itself is
running (I can ssh to it) and the daqd process is running but there's a red light for
it on the RFM screen and dataviewer won't connect to it.

If Alex isn't over by ~10 AM, we should call him and ask for help.
  355   Tue Mar 4 10:08:21 2008 robUpdateComputersgreen lights unreliable when c0daqctrl down

So far I've tried powering off the framebuilder, power-cycling the RAID (it was showing an error message about bad IDE channel #4), and rebooting the LSC (just for fun). When I reset the LSC, its green light on the RFM_NETWORK screen did not turn red, making all these lights suspect. The iscepics40m process is what controls these red/green lights, so maybe it's gone wonky. It appears to be running however, on c1dcuepics, and it also seems to be functioning correctly in other ways (it's communicating correctly with the LSC).

Update: Alex and Jay came by. The solution was to reset the c0daqctrl processor, which apparently was not done in Rana's rebooting spree. Or maybe it needed to be done last.
  358   Tue Mar 4 23:22:32 2008 robDAQComputersc1susvme1&2 rebooted

I found that some channels from c1susvme1 and c1susvme2 were not being recording by the DAQ (and were not showing up in DV). I rebooted these processors, which fix the problem. If you see other cases of this (signal exactly zero, but not a testpoint problem), just reboot the corresponding processor.
  364   Fri Mar 7 17:10:01 2008 Max JonesUpdateComputersNoise Budget work
Noise budget has been moved to the svn system. A checked out copy is in the directory caltech. From now on, I will try to use the work cycle as outlined in the svn manual. Changes made today include the following:

Details of the modifications made may be found on the svn system. Please let me know if anyone has a suggestion or concern. Thank you - Max.
ELOG V3.1.3-