40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 248 of 344  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  2948   Tue May 18 16:19:19 2010 josephbUpdateCDSWe have two new IO chassis

We have 2 new IO chassis with mounting rails and necessary boards for communicating to the computers.  Still need boards to talk to the ADCs, DACs, etc, but its a start.  These two IO chassis are currently in the lab, but not in their racks.

They will installed into 1X4 and 1Y5 tomorrow.  In addition to the boards, we need some cables, and the computers need the approriate real time operating systems setup.  I'm hoping to get Alex over sometime this week to help work on that.

  2953   Wed May 19 16:09:11 2010 josephbUpdateCDSRacks to small for IO Chassis rails

So I discovered the hard way that the racks are not standard width, when I was unable to place a new IO chassis into the racks with rails attached.  The IO chassis is narrow enough to fit through without the rails however. 

I've talked to Steve and we decided on having some shelves made.  I've asked Steve to get us 6.  1 for each end (2), 1 for SUS, 1 for LSC, 1 for IO, and 1 extra.

  2958   Thu May 20 13:12:28 2010 josephbUpdateCDSPreparations for testing lsc,lsp, scy,spy together

In /cvs/cds/caltech/target/fb modified:

master: cleaned up so only io1 (IO processor), LSC, LSP, SCY, SPY were listed, along with their associated tpchan files.

daqdrc: fixed "dcu_rate 9 = 32768" to "dcu_rate 9 = 65536" (since the IO processor is running at 64k)

Added "dcu_rate 21 = 16384" and "dcu_rate 22 = 16384"

Changed "set gds_server = "megatron" "megatron" "megatron" 9 "megatron" 10 "megatron" 11;" to

set gds_server = "megatron" "megatron" 9 9;

The above change was made after reading Rolf's Admin guide: http://lhocds.ligo-wa.caltech.edu:8000/40m/Upgrade_09/CDS?action=AttachFile&do=get&target=RCG_admin_guide.pdf
The set gds_server is simply telling which computer the gds daemons are running on, and we don't need to do it 5 times.

In /cvs/cds/caltech/gds/params modified:

testpoint.par: added C-node7 and C-node8 for SCY and SPY respectively.

  2989   Wed May 26 10:58:29 2010 josephbUpdateCDSNew RCG checkout for use with all machines plus some issues

Now that we have multiple machines we'd like to run the new front end code on, I'm finding it annoying to have to constantly copy files back and forth to have the latest models on different machines.  So I've come to the conclusion that Rana was right all along, and I should working somewhere in /cvs/cds/caltech which gets mounted by everyone. 

However, this leads to the svn problem: I.e. I need recent code checked out from the RCG repository, but our current /cvs/cds/caltech/cds/advLigo directory is covered by the 40m SVN.  So for the moment, I've checked out the advLigoRTS from https://redoubt.ligo-wa.caltech.edu/svn/advLigoRTS/trunk into /cvs/cds/caltech/cds/advLigoRTS.  This directory will be kept as up to date as I can keep it, both by running svn update to get Alex/Rolf's changes and on my end by keeping the new and updated models.  It will remain linked the RCG repository and not the 40m repository.  At some point a better solution is needed, but its the best I can come up with for now.

Also, because we are starting to compile on different machines sometimes, you may run into a problem where a code won't run on a different machine.  This can be fixed by commenting out some lines in the startup script.  Go to the /cvs/cds/caltech/scripts directory.  Then edit the associated startSYS file by commenting out the lines that look like:

if [ `hostname` != megatron ]; then
echo Cannot run `basename $0` on `hostname` computer
exit 1

Unfortunately, this gets reverted each time "make SYS" and "make install-SYS" gets run.

The other issue this leads to is that some machines don't have as many CPUs available as others.  For example our new thin 1U machines have only 4 dual cores (8 CPUs total).  This means the specific_cpu setting of any of the codes cannot be higher than 7 (cores being numbered 0 through 7).  Core 0 is reserved for the real time kernel, and Core 1 will be used on all machines for the IO processor. This leaves only cores 2 through 7 available for models to use which include LSC, LSP, SUS, SUP, SPY, SCY, SPX, SCX, OMC, OMP, OAF, OAP?, IOC, IOP.  Since there are more than 6 models, duplication in final production code of specific_cpus will be necessary.  Codes which are all running on Megatron at one point will have to be rebuilt with new specific_cpu values when run on the actual final machine.

  2990   Wed May 26 12:59:26 2010 josephbUpdateCDSCreated sus, sup, scx, spx models

I created the sus model, which is the suspension controller for ITMX, ITMY, BS, PRM, SRM.  I also created sup, which is the suspension plant model for those same optics.

Updated /cvs/cds/caltech/target/fb  master and daqdrc files to add SUS, SUP models.  Megatron's /etc/rc.d/rc.local file has been updated to include all the necessary models as well.

The suspension controller needs the Binary IO outputs need to be checked and corrected if wrong by changing the constant connected to the exclusive or gates. Right now its using the end suspension binary output values which may not be correct.

  3005   Fri May 28 10:44:47 2010 josephbUpdatePEMDAQ down


 Although trends are available, I am unable to get any full data from in the past (using DTT or DV). I started the FB's daqd process a few times, but no luck. 

I blame Joe's SimPlant monkeying from earlier today for lack of a better candidate. I checked and the frames are actually on the FB disk, so its something else.

 I tried running dataviewer and dtt this morning.  Dataviewer seemed to be working.  I was able to get trends, full data on a 2k channel (seismic channels) and full data on a 16k channel (C1:PEM-AUDIO_MIC1)  This was tried for a period 24 hours a go for a 10 minute stretch.

I also tried dtt and was able to get 2k and 16k channel data, for example C1:PEM-AUDIO_MIC1.  Was this problem fixed by someone last night or did time somehow fix it?

  3007   Fri May 28 11:35:33 2010 josephbUpdateCDSTaking a step backwards to get stuff running

I've modified the lsc.mdl and lsp.mdl files back to an older configuration, where we do not use an IO processor.  This seems to let things work for the time being on megatron while I try to figure out what the is wrong with the "correct" setup which includes the IO processor.

Basically I removed the adcSlave = 1 line in the cdsParameters block.

I've attached a screen shot of the desktop showing one filter bank in the LSP model passing its output correctly to a filter block in the LSC.  I also put in a quick test filter (an integrator) and you can see it got to 80 before I turned off the offset.

So far this is only running on megatron, not the new machine in the new Y end.

The models being use for this are located in /cvs/cds/caltech/cds/advLigoRTS/src/epics/simLink

Attachment 1: working_lsc_lsp.png
  3008   Fri May 28 13:17:05 2010 josephbUpdateCDSFixed problem with channel access on c1iscex

Talked with Alex and tracked down why the codes were not working on the new c1iscex finally.  The .bashrc and .cshrc files in /home/controls/ on c1iscex has the following lines:


This was interfering with channel access and preventing read and writes from working properly.  We simply commented them out. After logging out and back in, the things like ezcaread and write started working, and we were able to get the models passing data back and forth.

Next up, testing RFM communications between megatron on c1iscex.  To do this, I'd like to move Megatron down to 1Y3, and setup a firewall for it and c1iscex so I can test the frame builder and testpoints at the same time on both machines.

  3057   Tue Jun 8 20:52:25 2010 josephbUpdatePEMDAQ up (for the moment)

As a test, I did a remote reboot of both Megatron and c1iscex, to make sure there was no code running that might interfere with the dataviewer.  Megatron is behind a firewall, so I don't see how it could be interfering with the frame builder.  c1iscex was only running a test module from earlier today when I was testing the multi-filter matrix part.  No daqd or similar processes were running on this machine either, but it is not behind a firewall at the moment. 

Neither of these seemed to affect the lack of past data.  I note the error message from dataviewer was "read(); errno=9".

Going to the frame builder machine, I ran dmesg.  I get some disturbing messages from May 26th and June 7th. There are 6-7 of these pairs of lines for each of these days, spread over the course of about 30 minutes.

Jun 7 14:05:09 fb ufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full

Jun 7 14:11:14 fb last message repeated 19 times

There's also one:

Jun 7 13:35:14 fb syslogd: /usr/controls/main_daqd.log: No space left on device

I went to /usr/controls/ and looked at the file.  I couldn't read it with less, it errored with Value too large for defined data type.  Turns out the file was 2.3 G.  And had not been updated since June 7th.  There were also a bunch of core dump files from May 25th, and a few more recent.  However the ones from May 25th were somewhat large, half a gig each or so.  I decided to delete the main_daqd.log file as well as the core files.

This seems to have fixed the data history for the moment (at least with one 16k channel I tested quickly). However, I'm now investigating why that log file seems to have filled up, and see if we can prevent this in the future.


As before, I am unable to get data from the past. With DTT on Allegra I got data from now, but its unavailable from 1 hour ago. Same problem using mDV on mafalda. I blame Joe again - or the military industrial complex.




 Although trends are available, I am unable to get any full data from in the past (using DTT or DV). I started the FB's daqd process a few times, but no luck. 

I blame Joe's SimPlant monkeying from earlier today for lack of a better candidate. I checked and the frames are actually on the FB disk, so its something else.

 I tried running dataviewer and dtt this morning.  Dataviewer seemed to be working.  I was able to get trends, full data on a 2k channel (seismic channels) and full data on a 16k channel (C1:PEM-AUDIO_MIC1)  This was tried for a period 24 hours a go for a 10 minute stretch.

I also tried dtt and was able to get 2k and 16k channel data, for example C1:PEM-AUDIO_MIC1.  Was this problem fixed by someone last night or did time somehow fix it?


  3069   Fri Jun 11 15:04:25 2010 josephbUpdateCDSMulti-filter matrix medm screens finished and script for copying filters from SOS file

I've finished the MEDM portion of the RCG FiltMuxMatrix part.  Now it generates an appropriate medm screen for the matrix, with links to all the filter banks.  The filter bank .adl files are also generated, and placed in a sub directory with the name of the filter matrix as the name of the sub directory.

The input is the first number and the output is the second number.  This particular matrix has 5 inputs (0 through 4) and 15 outputs (0 through 14). Unfortunately, the filter names can't be longer than 24 characters, which forced me to use numbers instead of actual part names for the input and output.


The key to the numbers is:




AS11_I 1
AS11_Q 2
AS55_I 3
AS55_Q 4
POP11_I 6
POP11_Q 7
POP55_I 8
POP55_Q 9
REFL11_I 11
REFL11_Q 12
REFL55_I 13
REFL55_Q 14

To get this working required modifications to the feCodeGen.pl and the creation of mkfiltmatrix.pl (which was based off of mkmatrix.pl).  These are located in /cvs/cds/caltech/cds/advLigoRTS/src/epics/util/

In related news, I asked Valera if he could load the simulated plant filters he had generated, and after several tries, his answer was no.  He says it has the same format as those filter they pass to the feed forward banks down in Livingston, so he's not sure why they won't work.

I tested my script, FillFotonFilterMatrix.py, on some simple second order section filters (like gain of 1, with b1 = 1.01, b2 = 0.02, a1 = 1.03, a2 = 1.04), and it populated the foton filter file correctly, and was parsed fine by Foton itself.  So I'm going to claim the script is done and its the fault of the filters we're trying to load.  This script is now living in /cvs/cds/caltech/chans/ , along with a name file called lsp_dof2pd_mtrx.txt which tells the script that DARM is 0, CARM is 1, etc.  To run it, you also need a SOS.txt file with the filters to load, similar to the one Valera posted here, but preferably loadable.

I also updated my progess on the wiki, here.

Attachment 1: FiltMatrixMedm.png
  3080   Wed Jun 16 11:31:19 2010 josephbSummaryComputersRemoved scaling fonts from medm on Allegra

Because it was driving me crazy while working on the new medm screens for the simulated plant, I went and removed the aliased font entries in /usr/share/X11/fonts/misc/fonts.alias that are associated with medm.  Specifically I removed the lines  starting with widgetDM_.  I made a backup in the same directory called fonts.alias.bak with the old lines.

Medm now behaves the same on op440m, rosalba, and allegra - i.e. it can't find the widgetDM_ scalable fonts and defaults to a legible fixed font.

  3106   Wed Jun 23 15:15:53 2010 josephbSummaryComputers40m computer security issue from last night and this morning

The following is not 100% accurate, but represents my understanding of the events currently.  I'm trying to get a full description from Christian and will hopefully be able to update this information later today.


Last night around 7:30 pm, Caltech detected evidence of computer virus located behind a linksys router with mac address matching our NAT router, and at the IP  We did not initially recognize the mac address as the routers because the labeled mac address was off by a digit, so we were looking for another old router for awhile.  In addition, pings to were not working from inside or outside of the martian network, but the router was clearly working.  

However, about 5 minutes after Christian and Mike left, I found I could ping the address.  When I placed the address into a web browser, the address brought us to the control interface for our NAT router (but only from the martian side, from the outside world it wasn't possible to reach it).

They turned logging on the router (which had been off by default) and started monitoring the traffic for a short time.  Some unusual IP addresses showed up, and Mike said something about someone trying to IP spoof warning coming up.  Something about a file sharing port showing up was briefly mentioned as well.

The outside IP address was changed to and dhcp which apparently was on, was turned off.  The password was changed and is in the usual place we keep router passwords.

Update: Christian said Mike has written up a security report and that he'll talk to him tomorrow and forward the relevant information to me.  He notes there is possibly an infected laptop/workstation still at large.  This could also be a personal laptop that was accidently connected to the martian network.  Since it was found to be set to dhcp, its possible a laptop was connected to the wrong side and the user might not have realized this.


  3107   Wed Jun 23 15:33:42 2010 josephbUpdateCDSDaily Downs Update

I visited downs and announced that I would be showing up again until all the 40m hardware is delivered. 

I brought over 4 ADC boards and 5 DAC boards which slot into the IO chassis.

The DACs are General Standards Corporation, PMC66-16AO16-16-F0-OF, PCIe4-PMC-0 adapters.

The ADCs are General Standards Corporation, PMC66-16AI6455A-64-50M, PCIe4-PMC-0 adapters.

These new ones have been placed with the blue and gold adapter boards, under the table behind the 1Y4-1Y5 racks.

With the 1 ADC and 1 DAC we already have, we now have enough to populated the two ends and the SUS IO chassis.  We have sufficient Binary Output boards for the entire 40m setup.  I'm going back with a full itemized list of our current equipment, and bring back the remainder of the ADC/DAC boards we're due.  Apparently the ones which were bought for us are currently sitting in a test stand, so the ones I took today were from a different project, but they'll move the test stand ones to that project eventually.

I'm attempting to push them to finish testing the IO chassis and the remainder of those delivered as well.

I'd like to try setting up the SUS IO chassis and the related computer this week since we now have sufficient parts for it.  I'd also like to move megatron to 1Y3, to free up space to place the correct computer and IO chassis where its currently residing.

  3116   Thu Jun 24 16:59:24 2010 josephbUpdateVACFinished restoring the access connector and door

[Jenne,  Kiwamu, Steve, Sharmila, Katherine, Joe]

We finished bolting the door on the new ITMX (old ITMY) and putting the access connector section back into place.  We finished with torquing all the bolts to 40 foot-pounds.

  3119   Fri Jun 25 08:10:23 2010 josephbUpdateCDSDaily Downs Update

Yesterday afternoon I went to downs and acquired the following materials:

2 100 ft long blue fibers, for use with the timing system.  These need to be run from the timing switch in 1Y5/1Y6 area to the ends.

3 ADCs (PMC66-16AI6455A-64-50M) and 2 DACs (PMC66-16AO16-16-F0-OF), bringing our total of each to 8.

7 ADC adapter boards which go in the backs of the IO chassis, bringing our total for those (1 for each ADC) to 8.

There were no DAC adapter boards of the new style available.  Jay asked Todd to build those in the next day or two (this was on Thursday), so hopefully by Monday we will have those.

Jay pointed out there are different styles of the Blue and Gold adapter boxes (for ADCs to DB44/37) for example.  I'm re-examining the drawings of the system (although some drawings were never revised to the new system, so I'm trying to interpolate from the current system in some cases), to determine what adapter style and numbers we need.  In any case, those do not appear to have been finished yet (there basically stuffed boards in a bag in Jay's office which need to be put into the actual boxes with face plates).

When I asked Rolf if I could take my remaining IO chassis, there was some back and forth between him and Jay about numbers they have and need for their test stands, and having some more built.  He needs some, Jay needs some, and the 40m still needs 3.  Some more are being built.  Apparently when those are finished, I'll either get those, or the ones that were built for the 40m and are currently in test stands.



Aparently Friday afternoon (when we were all at Journal Club), Todd dropped off the 7 DAC adapter boards, so we have a full set of those.

Things still needed:

1) 3 IO chassis (2 Dolphin style for the LSC and IO, and 1 more small style for the South end station (new X)).  We already have the East end station (new Y) and SUS chassis.

2) 2 50+ meter Ethernet cables and a router for the DAQ system.  The Ethernet cables are to go from the end stations to 1Y5-ish, where the DAQ router will be located.

3) I still need to finish understanding the old drawings drawings to figure out what blue and gold adapter boxes are needed.  At most 6 ADC, 3 DAC are necessary but it may be less, and the styles need to be determined.

4) 1 more computer for the South end station.  If we're using Megatron as the new IO chassis, then we're set on computers.  If we're not using Megatron in the new CDS system, then we'll need a IO computer as well.  The answer to this tends to depend on if you ask Jay or Rolf.


  3124   Sat Jun 26 20:16:44 2010 josephbUpdateCDSNew checkout of RCG from SVN and changes made

ORPHAN ENTRY FOUND ON ROSALBA:::::::::::::::::::::::::::::::::::::::::::::::::::>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

We did svn update.  Then Alex realized he missed adding some files, so added them to the svn, and then we checked out again.


We rebuilt awg, fb, nds.

We reloaded service xinet.d, after editing /etc/xinetd.d/chnconf.  We changes all the tpchn_c1 to tpchn_c1SYS


There's a new naming scheme for the model files.  They start with the site name.  So, lsc becomes c1lsc. 


On any machine you want code running, make a symbolic link to /cvs/cds/rtcds/ in /opt/

  3127   Mon Jun 28 12:48:04 2010 josephbSummaryCDSCDS adapter board notes

The following is according to the drawing by Ben Abbott found at http://www.ligo.caltech.edu/~babbott/40m_sus_wiring.pdf

This applies to SUS:

Two ICS 110Bs.  Each has 2 (4 total) 44 shielded conductors going to DAQ Interface Chassis  (D990147-A).  See pages 2 and 4.

Three Pentek 6102 Analog Outputs to LSC Anti-Image Board (D000186 Rev A).  Each connected via 40 conductor ribbon cable (so 3 total). See page 5.

Eight XY220 to various whitening and dewhitening filters.  50 conductor ribbon cable for each (8 total). See page 10.

Three Pentek 6102 Analog Input to Op Lev interface board. 40 conductor ribbon cable for each (3 total).  See page 13.


The following look to be part of the AUX crate, and thus don't need replacement:

Five VMIC113A to various Coil Drives, Optical Levers, and Whitening boards.  64 conductor ribbon cable for each (5 total). See page 11.

Three XY220 to various Coil boards. 50 conductor ribbon for each (3 total).  See page 11.

The following is according to the drawing by Jay found at http://www.ligo.caltech.edu/~jay/drawings/d020006-03.pdf

This applies to WFS and LSC:

Two XY220 to whitening 1 and 2 boards.  50 conductor ribbon for each (2 total).  See page 3.

Pentek 6102 to LSC Anti-image. 50 conductor ribbon. (1 total). See page 5.


The following are unclear if they belong to the FE or the Aux crate.  Unable to check the physical setup at the moment.

One VMIC3113A to LSC I & Q, RFAM, QPD INT. 64 conductor ribbon cable. (Total 1).  See page 4.

One XY220 to QPD Int.  50 conductor ribbon cable. (Total 1). See page 4.


The following look to be part of WFS, and aren't needed:

Two Pentek 6102 Analog Input to WFS boards. 40 conductor ribbon cables (2 Total). See page 1.

The following are part of the Aux crate, and don't need to be replaced:

Two VMIC3113A to Demods, PD, MC servo amp, PZT driver, Anti-imaging board. 64 conductor ribbon cable (2 Total). See page 3.

Two XY220 to Demods, MC Servo Amp, QPD Int boards.  50 conductor ribbon cable (2 Total). See page 3.

Three VMIC4116 to Demod and whitening boards.  50 conductor ribbon cable (3 Total). See page 3.

  3134   Tue Jun 29 12:08:43 2010 josephbUpdateCDSDaily Downs Update

I talked with Rolf, and asked if we were using Megatron for IO.  The gist boiled down to we (the 40m) needed to use it for something, so yes, use it for the IO computer.  In regards to the other end station computer, he said he just needed a couple of days to make sure it doesn't have anything on it they need and to free it up.

I had a chat with Jay where he explained exactly what boards and cables we need.  Adapter boards are 95% of the way there.  I'll be stopping by this afternoon to collect the last few I need (my error this morning, not Jays).  However it looks like we're woefully short on cables and we'll have to make them. I also acquired 2 D080281 (Dsub 44 x2 to SCSI).

For each 2 Pentek DACs plus a 110B, you need 1 DAC adapter board (D080303 with 2 connectors for IDC40 and a SCSI).  You also need a D080281 to plug onto the back of the Sander box (going to the 110Bs) to convert the D-sub 44 pins to SCSI.

LSC will need none, SUS will need 3, IO will need 1, and the ends will need 1 each.  We have a total of 6, we're set on D080303s.  We have 3 110Bs, so we need one more D080281 (Dsub44 to SCSI).  I'll get that this afternoon.

For each XVME220, we'll need one D080478 binary adapter.  We have 8 XVME220s, and we have 8 boards, so we're set on D08478s.

For the ends, there's a special ADC to DB44/37 adapter, which we only have 1 one of.  I need to get them to make 1 more of these boxes.

We have 1 ADC to DB37 adapter, of which we'll need  1 more of as well, one for IO and one for SUS. 

However, for each Pentek ADC, we need a IDC40 to DB37 cable.  For each Pentek DAC we need an IDC40 to IDC40 cable.  We need a SCSI cable for each 110B.  I believe the current XVME220 cables plug directly in the BIO adapter boxes, so those are set.

So we need to make or acquire 11 IDC40 to DB37 cables, 7 IDC40 to IDC40 cables, and 3 SCSI cables.


Summary Needed:

1 ADC to DB44/37 for the End (D080397)

1 ADC adapter (D080302)

1 Dsub44 to SCSI (D080291)

11 IDC40 to DB37 cables

7 IDC40 to IDC40 calbes

3 SCSI cables

PLUS from before:

3 IO Chassis (2 Dolphin, 1 Small)

1 1U computer (8 core for end)

Router/2 50+m ethernet for DAQ

  3136   Tue Jun 29 14:19:44 2010 josephbUpdateCDSDaily Downs Update (Part 2)

I picked up the ribbon cable connectors from Jay.  It looks like we'll have to make the new cables for connecting the ADCs/DACs myself (or maybe with some help).  We should be able to make enough ribbon cables for use now.  However,  I'm adding "Make nice shielded cables" to my long term to do list.

I pointed out the 2 missing adapter boxes we need to Jay.  He has the parts (I saw them) and will try to get someone to put it together in the next day or so.  I also picked up 2 more D080281 (DB44 to SCSI), giving us enough of those.

I once again asked Jay for an update on IO chassis, and expressed concern that without them the CDS effort can't really go forward, and that we really need this to come together ASAP.  He said they still need to make 3 new ones for us.

So we're still waiting on a computer, 3 IO chassis, router + ethernet.

  3144   Wed Jun 30 12:01:20 2010 josephbUpdateCDSSUS IO Chassis

I spent this morning populating the SUS IO Chassis and getting it ready for installation into the 1Y4 rack.  I discovered I'm lacking the internal cables to connect the ADC/DAC boards to the AdL adapter boards that also go in the chassis (D0902006 and D0902496).  I'm not sure where Alex got the cables he used for the end IO chassis he had put in.  I'll be going to Downs after the meeting today either to get cables or parts for cables, and get the SUS chassis wired up fully.

I'd also like to confirm with Alex that the OSS-MAX-EXP-ELB-C board that goes in the IO chassis matches the host interface board that goes in the computer (OSS-HIB2-PE1x4-1x4 Re-driver HIB, since we spent half a day the last time we installed an IO chassis determining that one of the pair was bad or didn't match.


The SUS chassis has been populated in the following way:

Treton Board

Slot 1 ADC PMC66-16AI6455A-64-50M

Slot 2 DAC PMC66-16AO16-16-F0-OF 

Slot 3-6 BO Contec DIO-1616L-PE Isolated Digital IO board

Slot 7 ADC PMC66-16AI6455A-64-50M

Slot 8-9 DAC PMC66-16AO16-16-F0-OF 

Back  Board

Slot 1 ADC adapter D0902006

Slot 2 DAC adapter D0902496-v1

Slot 7 ADC adapter D0902006

Slot 8-9 DAC adapter D0902496-v1

  3153   Thu Jul 1 14:03:40 2010 josephbUpdatePEMTemporary disconnect of some PEM channels for ~20 minutes

In order to identify the output adapter of the BNC patch panel used for about 20 PEM channels, I had to disconnect its power and remove the back panel.  Channels coming into that panel (seismometers and so forth) was out from 1:36 to 1:56 pm.

I did a quick check of some of the channels and it looks like its working again after putting it all back together.

  3157   Fri Jul 2 11:33:15 2010 josephbUpdateCDSc1sus needs real time linux to be setup on it

I connected a monitor and keyboard to the new c1sus machine and discovered its not running RTL linux.  I changed the root password to the usual, however, without help from Alex I don't know where to get the right version or how to install it, since it doesn't seem to have an obvious CD rom drive or the like.  Hopefully Tuesday I can get Alex to come over and help with the setup of it, and the other 1-2 IO chassis.

  3158   Tue Jul 6 11:57:06 2010 josephbUpdateCDSDaily Downs Update

I went to talk to Rolf and Jay this morning.  I asked Rolf if a chassis was available, so he went over and disconnected one of his test stand chassis and gave it to me.  It comes with a Contect DIO-1616L-PE Isolated Digital IO board and an OSS-MAX-EXP-ELB-C, which is a host interface board.  The OSS board means it has to go into the south end station.  There's a very short maximum cable length associated with that style, and the LSC and IOO chassis will be further than that from their computers (we have dolphin connectors on optical fiber for those connections).

I also asked Jay for another 4 port 37 d-sub ADC blue and gold adapter box, and he gave me the pieces.  While over there, I took 2 flat back panels and punched them with approriate holes for the scsi connectors that I need to put in them.  I still need to drill 4 holes in two chassis to mount the boards, and then a bit of screwing.  Shouldn't take more than an hour to put them both together.  At that point, we should have all the adapter boxes necessary for the base design.  We still need some stuff for the green locking, as noted on Friday.

Major hardware still needed:

2 Dolphin style IO chassis

1 computer for south end front end

  3160   Tue Jul 6 17:07:56 2010 josephbUpdateCDSc1sus status

I talked to Alex, and he explained the steps necessary to get the real time linux kernel installed.  It basically went like copy the files from c1iscex (the one he installed last month) in the directory /opt/rtlidk-2.2 to the c1sus locally.  Then go into rtlinux_kernel_2_6, and run make and make install (or something like that - need to look at the make file).  Then edit the grub loader file to look like the one on c1iscex (located at /boot/grub/menu.lst).

This will then hopefully let us try out the RCG code on c1sus and see if it works.

  3176   Thu Jul 8 14:11:16 2010 josephbUpdateComputersSome channels not being recorded!!!

This has been fixed, thanks to some help from Alex. It doesn't correspond to new computers being put in, but rather corresponds to a dcu_id change I had made in the new LSC model.

The fundamental problem is way back when I built the new LSC model using "lsc" as the name instead of something like "tst", I forgot to go to the current frame builder master file (/cvs/cds/caltech/chans/daq/master) and comment out the C1LSC.ini line. Initially there was no conflict with c1susvme, because the initially was dcu_id 13. The dcu_id was eventually changed to 10 from 13 , and thats when it conflicted with the c1susvme2 dcu_id which was also 10. I checked it against wiki edits to my dcu_id list page and I apparently updated the list on May 20th when it changed from 13 to 10, so the time frame fits.  Apparently it was previously conflicting with C0GDS.ini or C1EXC.ini, which both seem to have dcu_id = 13 set, although the C1EXC file is all commented out. The C0GDS.ini file seems to be LSC and ASC test points only.

The solution was to comment out the C1LSC.ini file line in the /cvs/cds/caltech/chans/daq/master file and restart the framebuilder with the fixed file.


[Rana, Jenne]

We discovered to our great dismay that several important channels (namely C1:IOO-MC_L, but also everything on c1susvme2) are not being recorded, and haven't been since May 17th.  This corresponds to the same day that some other upgrade computers were installed.  Coincidence?

We've rebooted pretty much every FE computer and the FrameBuilder and DAQ_CONTROL approximately 18 times each (plus or minus some number).  No matter what we do, or what channels we comment out of the C1SUS2.ini file, we get a Status on the DAQ_Detail screen for c1susvme2 of 0x1000.  Except sometimes it is 0x2000.  Anyhow, it's bad, and we can't make it good again. 

I have emailed Joe about fixing this (with some assistance from Alberto, since we all know how much he likes doing the Nuclear Reboot option for the computers :)


  3177   Thu Jul 8 14:32:42 2010 josephbUpdateCDSDaily Downs Update

After talking with Rolf, and clarifying exactly which machine I wanted, he gave me an 4600 Sun machine (similar to our current megatron).  I'm currently trying to find a good final place for it, but its at least here at the 40m. 

I also acquired 3 boards to plug our current VMIPMC 5565 RFM cards into, so they can be installed in the IO chassis.  These require +/- 5V power be connected to the top of the RFM board, which would be not possible in the 1U computers, so they have to go in the chassis.  These style boards prevent the top of the chassis from being put on (not that Rolf or Jay have given me tops for the chassis).  I'm planning on using the RFM cards from the East End FE, the LSC FE, and the ASC FE. 

I talked to Jay, and offered to upgrade the old megatron IO chassis myself if that would speed things up.  They have most of the parts, the only question being if Rolf has an extra timing board to put in it.  Todd is putting together a set of instructions on how to put the IO chassis together and he said he'd give me a copy tomorrow or Monday.  I'm currently planning on assembling it on Monday.  At that point I only need 1 more IO chassis from Rolf.



When I asked about the dolphin IO chassis, he said we're not planning on using dolphin connections between the chassis and computer anymore.  Apparently there was some long distance telecon with the dolphin people and they said the Dolphin IO chassis connection and RFM doesn't  well together (or something like that - it wasn't very clear from Rolf's description).  Anyways, the other style apparently is now made in a fiber connected version (they weren't a year ago apparently), so he's ordered one.  When I asked why only 1 and what about the IOO computer and chassis, he said that would either require moving the computer/chassis closer or getting another fiber connection (not cheap).

So the current thought I hashed out with Rolf briefly was:

We use one of the thin 1U computers and place that in the 1Y2 rack, to become the IOO machine.  This lets us avoid needing a fiber.  Megatron becomes the LSC/OAF machine, either staying in 1Y3 or possibly moving to 1Y4 depending on the maximum length of dolphin connection because LSC and the SUS machine are still supposed to be connected via the Dolphin switch, to test that topology.

I'm currently working on an update to my CDS diagram with these changes and will attach it to this post later today.

  3185   Fri Jul 9 11:09:14 2010 josephbUpdateComputersFb40m and a few other machines turned off briefly just before 11am

I turned off fb40m2 and fb40m temporarily while we added an extra power strip  to the (new) 1X6 rack at the bottom in the back.  This is to allow for the addition of the 4600 computer  given to us by Rolf (which needs a good name) into the rack above the fb machine.  The fb40m2 was unfortunately plugged into the main power connectors, so we unplugged two of its cables, and put them into the new strip. While trying to undo some of the rats nest of cables in the back I also powered down and unpluged briefly the c0dcu1, the pem crate, and the myrinet bypass box.

I am in the process of bringing those machines back up and restoring the network.

Also this morning, Megatron was moved from the end station into the (new) 1X3 rack, along with its router.  This is to allow for the installation of the new end computer and IO chassis.


  3213   Wed Jul 14 10:00:14 2010 josephbUpdatePhase CameraWork near 1Y2 yesterday

Razib and I were attempting to get the output of a photodiode (PD55A in this case) recorded, so that we could independently measure the slow (~1-10 Hz) fluctuations of the light incident on the camera.  This would then allow us to subtract those fluctuations out, letting us get at the camera noise in the case with signal present (as opposed to just a dark noise measurement when we look at the noise with no signal present).

Originally I was thinking of using one empty patch panel BNCs used for PEM channels down by the 1Y7 rack and go through a 110B, although Alberto pointed out he had recently removed some monitoring equipment, which watched the amplitude modulation at various frequencies of the RF distribution (i.e. 33 MHz, etc).  This equipment output a DC voltage proportional to the amplitude of the RF signals.  The associated channel names were C1:IOO-RFAMPD_33MHZ, C1:IOO-RFAMPD_33MHZ_CAL, C1:IOO-RFAMPD_133MHZ, etc.  These are slow channels, so I presume they enter in via the slow computers, probably via pentek (I didn't check that, although in hindsight I probably should have taken the time to find exactly where they enter the system).  The connections them selves were a set of BNCs on the south side, half way up the 1Y2 rack.

We simply chose one, the 33 MHz channel in this case, and connected.  At around this time, the MC did become unlocked, although it looked like it was due to the MC2 watchdog tripping.  The initial theory was we had bumped the Mode Cleaner while looking around for some BNC cables, although from what Rana had to do last night, it probably was the connection.  We were able to restore the watchdog and confirm that the optic started to settle down again.  Unfortunately, I had to leave about 5 minutes later, and didn't do as thorough an investigation as was warranted.

  3216   Wed Jul 14 11:54:33 2010 josephbUpdateDAQDebugging Guralp and reboots

This is regards to zero signal being reported by the channels C1:PEM-SEIS_GUR1_X, C1:PEM-SEIS_GUR1_Y, and C1:PEM-SEIS_GUR1_Z.

I briefly swapped Guralp 1 EW and Guralp 2 EW to confirm to myself that it was not on the gurlap end (although the fact that its digital zero is highly indicative a digital realm problem).  I then unplugged the 17-32, and then 1-16 channel connections to the 110B.  I saw floating noise on the GUR2 channels, but still digital zero on the GUR1 channels, which means its not the BNC break out box.

There was a spare 110B, unconnected in the crate, so to do a quick test of the 110B, I turned off the crate and swapped the 110Bs, after copying the switch configuration of the first 110B to the second one.  The original 110B was labeled ADC 1, while the second 110B was labeled ADC 0.  The switches were identical except for the ones closest to the Dsub connectors on the front.  All those switches in that set were to the right, when looking down at the switches and the Dsub connectors pointing towards yourself.

Unfortunately, the c0duc1 never seemed to come up with the new 110B (ADC 0).  So we put the original 110B back.  And turned the crate back on. 

The fb then didn't seem to come back quite right.  We tried rebooting fb40m it, but its still red with status 1.  c0daqctrl is green, but c0dcu1 is red, although I'm not positive if thats due to fb40m being in a strange state.  Jenne tried a telnet in to port 8087 and shutdown, but that didn't seem to help.  At this point, we're going to contact Alex when he gets in around 12:30.


  3226   Thu Jul 15 11:58:50 2010 josephbUpdateComputersAdded channel to ADCU_PEM (C0DCU1)

I modified the C1ADCU_PEM.ini file in /cvs/cds/caltech/chans/daq/ (after making a backup), and added a temporary channel called C1:PEM-TEMP_9, the 9 corresponding to the labeled 9 channel on the front of the BNC breakout in the 1Y7 rack.  The chnnum it was set to is 15008 (it was commented out and called C1:PEM-PETER_FE).  I also set the data rate to 2048.

I then did telnet fb40m 8087, and shutdown, and also hit the blue reconfig button on the DAQ status screen for the C0DCU1 machine.  The framebuilder came back up.  I confirmed the temporary channel, as well as the Guralp channels were still working from C0DCU1.

We have strung a cable in the cable trays from the SP table to the 1Y7 rack, which has been labeled as "Phasecam PD".  This will be used to record the output of an additional photodiode.


  3238   Fri Jul 16 16:07:14 2010 josephbUpdateComputersPossible solution for the last ADC

After talking with Jenne, I realized the ADC card in the c1ass machine was currently going unused.  As we are short an ADC card, a possible solution is to press that card into service.  Unfortunately, its currently on a PMC to PCI adapter, rather than PMC to PCIe adapter.  The two options I have are to try to find a different adapter board (I was handed 3 for RFM cards, so its possible there's another spare over in downs - unfortunately I missed Jay when I went over at 2:30 to check).  The other option is put it directly into a computer, the only option being megatron, as the other machines don't have full length PCI slot. 

I'm still waiting to hear back from Alex (who is in Germany for the next 10 days) whether I can connect both in the computer as well as with the IO chassis.

So to that end, I briefly turned off the c1ass machine, and pulled the card.  I then turned it back on, restarted all the code as per the wiki instructions, and had Jenne go over how it looked with me, to make sure everything was ok.

There is something odd with some of the channels reading 1e20 from the RFM network.  I believe this is related to those particular channels not being refreshed by their source (which is other suspension front end machines), so its just sitting at a default until the channel value actually changes.



  3273   Fri Jul 23 08:18:03 2010 josephbUpdateCDSNot enough room in IO chassis for RFM card - need to swap PMC to PCIe adapters

The End IO chassis have small trenton boards, which apparently only have 5 usuable PCI slots, even though there are 6 on the board.  This is because of the way the the host interface board is setup and its closeness to the 2nd to last PCI slot

The PMC to PCIe adapters I was handed by Jay for use with the RFM cards require a 4 pin power connection at the top, which are not available inside the thin 1U computers.

The only solution I can come up with is swap the PMC to PCIe adapters for the RFM cards with adapters for some of the already installed ADCs and DACs which do not require power directly from the power supply.  This should make it possible to mount the RFM card in the computer, at least for the ends.  Since the SUS and IOO chassis will have more slots available than needed, the RFM cards can be slotted into those.  The SUS has to fit in the chassis since the computer will have the Infiband host adapter and a dolphin connector for talking to the LSC machine.

There is still the problem of actually getting the RFM card into the computer, but that should be possible with a little bit of bending of the left side of the computer frame.

  3274   Fri Jul 23 10:16:44 2010 josephbUpdateCDSRerouted RFM around c1lsc, took RFM card out of c1lsc

 I re-routed around the c1lsc machine this morning.  I turned the crate off, and disconnected the transmission fiber from  c1lsc (which went to the receiver on c1asc).  I then took the receiving fiber from c1lsc and plugged it into the receiver on c1asc. 

I pulled out the c1lsc computer from the VME crate and pulled out the RFM card, which I needed for the CDS upgrade.  I then replaced the lsc card back in the crate and turned it back on.  Since there hasn't been a working version of the LSC code on linux1 since I overwrote it with the new CDS lsc code, this shouldn't have any significant impact on the interferometer.

I've confirmed that the RFM network seems to be in a good state (the only red lights on the RFM timing and status medm screen are LSC, ASC, and ETMX).  Fast channels can still be seen with dataviewer and fb40m appears to still be happy.

The RFM card has found its new home in the SUS IO Chassis.  The short fiber that used to go between c1asc and c1lsc is now on the top shelf of the new 1X3 rack.

  3275   Fri Jul 23 12:33:51 2010 josephbUpdateCDSmegatron, c1iscex, c1sus firewalled

Kiwamu and I strung an ethernet cable from the new 1X7 rack to the 1X3 rack. The cable is labeled c1iscex-daq on both ends.  This cable will eventually connect c1iscex's second ethernet port to the daq router.  However, for today, it plugged into the primary ethernet port and is going to a linksys router.  This is the same linksys router we used to firewall megatron.

The idea is to place megatron, c1sus, and c1iscex behind the firewall to prevent any problems with the currently running while doing RFM nework tests.

The way to get into the firewalled sub-network is to ssh into megatron.  The router will forward the ssh to megatron.  Inside the network, the computers will have the following IPs.  Router is, megatron is, c1sus is, and c1iscex is



  3289   Mon Jul 26 10:02:42 2010 josephbUpdateCDSRerouted RFM around c1lsc, took RFM card out of c1lsc

If you're refering to just the medm screen,  those can be restored from the SVN.  As we're moving to a new directory structure, starting with /opt/rtcds/caltech/c1/, the old LSC screens can all be put back in the /cvs/cds/caltech/medm/c1/lsc directory if desired.

The slow lsc aux crate, c1iscaux2, is still working, and those channels are still available.  I confirmed that one was still updating. As a quick test, I went to the SVN and pulled out the C1LSC_RFADJUST.adl file, renamed it to C1LSC_RFadjust.adl and placed it in /cvs/cds/caltech/medm/c1/lsc/, and checked it linked properly from the C1IOO_ModeCleaner.adl file.  I haven't touched the modulation depths, as I didn't want to mess with the mode cleaner, but if I get an OK, we can test that today and confirm that modulation depth control is still working.


 I just realized that an unfortunate casualty of this LSC work was the deletion of the slow controls for the LSC which we still use (some sort of AUX processor). For example, the modulation

depth slider for the MC is now in an unknown state.


  3296   Tue Jul 27 11:24:53 2010 josephbHowToComputer Scripts / Programskilldataviewer script

I placed a script for killing all instances of the dataviewer program on the current computer in /cvs/cds/caltech/scripts/general/.  Its called killdataviewer.  This is intended to get rid of a bunch of zombie dataviewer processes quickly.  These processes get into this bad state when the dataviewer program is closed in any way other than the graphical menu File -> Exit option.

Its contents are very simple:


kill `ps -ef | grep dataviewer | grep -v grep | grep -v killdataviewer | awk '{print $2}'`

  3319   Thu Jul 29 12:31:24 2010 josephbUpdateCDSWorking DAC, working IOP - next up SUS

Ok, after a few minutes of talking to Alex, I got the correct "GUI syntax" through my head, and we now have a simple working green end control which in fact puts signals out through the DAC.

Note to self, do not put any additional filters or controls in the IOP module.  Basically just change the master block with GDS numbers, DCU_ID numbers, etc.  When using a control model, copy the approriate ADC and ADC selector or DAC to the control model.  It will magically be connected to the IOP.

A correct example of a simple control model is attached.

Next in line is to get the adapter boxes for SUS into the new 1X5 rack and get started on SUS filter conversion and figuring out which ADC/DAC channels correspond to which inputs.


Attachment 1: Simple_Green_Control.png
  3342   Sat Jul 31 17:37:36 2010 josephbUpdateCDSCables needed for CDS test

Last Thursday, Kiwamu and I went through the cabling necessary for a full damping test of the vertex optics controled by the sus subsytem, i.e. BS, ITMX, ITMY, PRM, SRM.  The sus IO chassis is sitting in the middle of the 1X4 rack.  The c1sus computer is the top 1U computer in that rack.


The hardest part is placing the 2x D-sub connectors to scsi on the lemo break out boxes connected to the 110Bs.  The breakout boxes can be seen at the very top of the picture Kiwamu took here.  These will require a minor modification to the back panel to allow the scsi cable to get out.  There are two of these boxes in the new 1X5 rack.  These would be connected by scsi to the ADC adapters in the back of the sus IO chassis in 1X4.  The connectors are currently behind the new 1X5 rack (along with some spare ADCs/DACs/BOs.

There are 3 cables going from 40 IDC to 37 D-sub (the last 3 wires are not used and do not need to be connected, i.e. 38-40).  These plug into the blue and gold ADC adapter box, the top one shown here.  There is one spare connection which will remain unused for the moment.  The 40 IPC ends plug into the Optical Lever PD boxes in the upper right of the new 1X4 rack (as seen in the top picture here - the boards on the right). At the back of the blue and gold adapter box is a scsi adapter which goes to the back of the IO chassis and plugs into an ADC.

In the back of the IO chassis is a 4th ADC which can be left unconnected at this point.  It will eventually be plugged into the BNC breakout box for PEM signals over in the new 1X7 rack, but is unneeded for a sus test.


There are 5 cables going from 3 SOS dewhite/anti-image boards and 2 LSC anti-image boards into 3 blue and gold DAC adapter boxes.  Currently they plug into the Pentek DACs at the bottom of the new 1X4 rack.  Ideally we should be able to simply unplug these from the Pentek DACs and plug them directly into the blue and gold adapter boxes.  However at the time we checked, it was unclear if they would reach.  So its possible new cables may need to be made (or 40 pin IDC extenders made). These boxes are then connected to the back of the IO chassis by SCSI cables.  One of the DAC outputs will be left unconnected for now.

Binary Output:

The Binary output adapter boxes are plugged into the IO chassis BO cards via D-sub 37 cables.  Note one has to go past the ADC/DAC adapter board in the back of IO chassis and plug directly into the Binary Output cards in the middle of the chassis.  The 50 pin IDC cables should be unplugged from XY220s and plugged into the BO adapter boxes.  It is unclear if these will reach.


We have a short fiber cable (sitting on the top shelf of the new 1X3 rack) which we can plug into the master timing distribution (blue box located in the new 1X6 rack) and into the front of the SUS IO chassis.  It doesn't quite make it going through all the holes at the top of the racks and through the cabling trays, so I generally only plug it in for actual tests.

The IO chassis is already plugged into the c1sus chassis with an Infiniband cable.


So in Summary to plug everything in for a SUS test requires:

  • 6x SCSI cables (3 ADC, 3 DAC) (several near bottom of new 1X3 rack)
  • 4x 37 D-sub to 37 D-sub connector (end connectors can be found behind new 1X5/1X6 area with the IO chassis stuff - Need to be made) (4 BO)
  • 3x 40 IDC to 37 D-sub connectors (end connectors can be found behind new 1X5/1X6 area - Need to be made)(ADC)
  • 5x 64 pin ribbon to 40 IDC cable (already exist, unclear if they will reach) (DAC)
  • 8x 50 pin IDC ribbon (already exist, unclear if they will reach) (BO)
  • 1x Double fiber from timing master to timing card
  • 1x Infiniband cable (already plugged in)

Tomorrow, I will finish up a channel numbering plan I started with Kiwamu on Thursday and place it in the wiki and elog.  This is for knowing which ADC/DAC/BO channel numbers correspond to which signals.  Which ADCs/DACs/BOs the cables plug into matter for the actual control model, otherwise you'll be sending signals to the wrong destinations.

WARNING: The channel numbers on the front Binary Output blue and gold adapter boxes are labeled incorrectly.  Channels 1-16 are really in the middle, and 17-32 are on the left when looking at the front of the box.  The "To Binary IO Module" is correct.

  3399   Wed Aug 11 13:28:44 2010 josephbUpdateCDSNew cdsIPCx parts, old part breaks models

While working on a test stand at LLO, I've discovered that there's a new update to the Real Time Code Generator that breaks virtually all of our current models.

Previously, we've been using the cdsIPCx part as a flexible link between models, and could be set to RFM (reflected memory), SHMEM (shared memory on the local computer), or PCIE (pci express or dolphin I think) depending on what was written in the IPC file (found in /cvs/cds/caltech/chans/ipc/C1.ipc).

Recently, three new parts were added called cdsIPCx_SHMEM, cdsIPCx_RFM, cdsIPCx_PCIE, and the previous cdsIPCx broken.  The cdsIPCx_***** has to match the type written in the IPC file or else it errors out with a rather unhelpful error message which doesn't tell you which part/line is in conflict...


***ERROR: IPCx type mis-match: I vs. ISHM
make: *** [g1lsc] Error 255

In anycase, when using a current checkout (updated or checked out after ~July 30th), all cdsIPCx blocks in the simulink models needs to be changed to the approriate type.  I'm trying to get a new CDS_PARTS.mdl file I updated into the svn, but for now you can just open up the model file directly in the /advLigoRTS/src/epics/simLink/lib directory and copy it in from there (i.e. cdsIPCx_SHMEM.mdl) to your model file.  I'm also trying to get the feCodeGen.pl file changed so the error message actually tells you what channel/part is mismatching so you don't have to guess.

An easy way to modify a file for all shared memory connections without doing a ton of cutting, pasting and renaming is:

sed -i 's/"cdsIPCx"/"cdsIPCx_SHMEM"/g' file_name_here.mdl

  3408   Thu Aug 12 14:01:53 2010 josephbUpdateCDScurrent status

First, awesome progress.

Second, the Binary Output parts do in fact need to be added to the c1sus model.  They don't need to appear in the IOP though.  (They are somehow automatically taken care of by the IOP code). 

It looks like in the 2004 revision in SVN, the cdsCDO32 part (located in the CDS_PARTS.mdl/IO_PARTS) was broken.  I fixed it and updated the svn with it, so a svn update should pull the corrected version.

I'm not sure whats wrong with the DAQs.  I'll try to take a closer look at the model file tonight and see if I can make suggestions.  When you write outputs on DAC_0 or DAC_1 is the C1SUS GDS TP screen showing anything?

  3440   Thu Aug 19 09:51:43 2010 josephbUpdateelogelog dead again

I found the elog dead again this morning, and the script didn't kill again. I modified the script to use the following line instead of "pkill elogd":

kill `ps -ef | grep elogd | grep -v grep | grep -v start-elog-nodus | awk '{print $2}'`

Hopefully the script will be a bit more robust in the future.



The elog was so dead this time that the restart script didn't work.  I followed the restart-by-hand instructions instead, with success.

 Just for added interest, I tried a different method when the restart script broke.  The "start-elog-nodus" script has a line "kill elogd".  This seems not to be actually killing anything anymore, which means the elog can't restart.  So this time I went for "kill <pid number>", and then ran the startup script.  This worked.  So it's the "kill elogd" which isn't working reliably.


  3446   Fri Aug 20 13:09:53 2010 josephbUpdateelogRebooted elog

I had to restart the elog again.

At this point, I'm going to try to get one of the GC guys to install gdb on nodus, and run the elog in the debugger, that way when it crashes the next time, I have some error output I can send back to the developer and ask why its crashing there.

  3464   Tue Aug 24 14:29:18 2010 josephbUpdateelogElog down for 1 minute

I'm going to take the elog down for one minute and restart it under gdb (using a copy of gdb stolen from fb40m since I couldn't figure out how to install an old enough version on nodus from source).  The terminal with information is running on Rosalba under the "Phase Noise" panel, so please don't close it.  Ideally, the next time the elog crashes, I'll have some output indicating why or at least the line in the code.  I can then look at the raw source code or send the line back to the developer and see if he has any ideas.

  3467   Wed Aug 25 12:18:47 2010 josephbUpdateelogTrying new version of elog to see if it helps stability

So unfortunately, I made the start-elog-nodus script smart enough to kill the debugging run I had (although thats probably good since there might have been issues with continuing to run - just poor timing on part of the crash).

In related news, I have gotten the latest version of the elog code to actually compile on Nodus.  I had to hack the cryptic.c file (elog/src/cryptic.c) to get it to work though.

The following was copied from the #ifdef _MSC_VER section of the code into the #else directly following that section. 

#define MAX(x,y) ((x)>(y)?(x):(y))
#define MIN(x,y) ((x)<(y)?(x):(y))
#define __alignof__(x) sizeof(x)
#define alloca(x) malloc(x)
#define mempcpy(d, s, n) ((char *)memcpy(d,s,n)+n)
#define ERANGE 34

I also removed #include <stdint.h> as the functionality it provides is covered by inttypes.h on Solaris machines, which is automatically included.

This new code was released August 5th 2010, while the old elog code we were running was 2.7.5 and was released sometime in 2008.  There are several crash fixes mentioned in the version notes so I'm hoping this may improve stability. I'm in the process of making a copy of the elog logbooks into the elog-2.8.0 install (so as to have a backup with the original 2.7.5).  I'm also copying over all the configuration files.   In a few minutes I'm going to try switching over to the new elog.  If it doesn't work, or is worse, its easy enough to just start up the current version.

All files are located in /cvs/cds/caltech/elog/elog-2.8.0 (the old directory is elog-2.7.5).  I've made  a new startup script called start-elog-nodus-2.8.0.  To start the new one, just run that script.  To start the old one, just go to the elog-2.7.5 directory and run the old start-elog-nodus script.

  3468   Wed Aug 25 12:40:28 2010 josephbUpdateelogReverted back to 2.7.5 until further testing is done

So apparently the themes/configurations didn't work so nicely on some of the logbooks with 2.8.0, so I'm reverting to 2.7.5 until I can figure out (assuming I can) how to get them to display properly.

  3471   Wed Aug 25 15:55:33 2010 josephbUpdateelogStaying with 2.7.5 until passwords sorted out

Turns out the elog version 2.8.0 uses a different encryption method than 2.7.5.  This mean the encrypted passwords stored in the elogd.cfg don't work with the new code.  elogd includes functionality to generate encrypted passwords, but unfortunately I don't know the administration passwords for some of the logbooks.  So I'm going to leave 2.7.5 running until I can get those added properly to the 2.8.0 cfg file.

  3473   Thu Aug 26 13:08:03 2010 josephbUpdateCDSWatch dogs for Vertex optics turned off

We are in the process of doing a damping test with the real time code and have turned off the vertex optics watchdogs temporarily, including BS, ITMs, SRM, PRM, MCs.

  3498   Tue Aug 31 16:42:26 2010 josephbUpdateCDSTemporarily reverting to CDS revision 2005

Apparently updating to the latest revision of the RCG has some issues with diaggui and awgtpman.  Alex had to do some recompiling up at Hanford which apparently took him some time.  He'll be coming by tomorrow to try to bring those codes to the front end machines.

As a temporary fix, until Alex gets here tomorrow, we're reverting to the 2005 revision in the svn of the cds code.  I'm placing it in the location it is supposed to go in the new advLIGO scheme, which is /opt/rtcds/caltech/c1/core/, which is where Keith had it at LLO.  Once we get the new codes working, we will do an svn update on that location and migrate our work to that install location, at which point I'll remove the old /cvs/cds/caltech/cds/advLigoRTS/ location.

  3507   Wed Sep 1 12:24:47 2010 josephbUpdateCDSTrying to get up to date CDS code runnning

Alex, Joe:

We copied the latest x02 to c1x02 and modified to our the config block in it.

We removed gds_node_id.  We just have one number now, the dcuid, which is unique for each controller, simulated plant and IOP.  Set site to C1 and host to c1sus.

Alex made the latest awgtpman backwards compatible, and checked that into svn.

We installed the latest framecpp onto c1sus from www.ldas-sc.ligo.caltech.edu/packages/ using wget.

wget www.ldas-sc.ligo.caltech.edu/packages/framecpp-1.18 and then used make.

This let us compile diagd on c1sus, using the command make stand in the /advLigoRTS/build area.

We copied gds from the seiteststand over at handford and are trying to build that on megatron.  However, there's a bunch of packages we need for it to install properly.  Alex said he'd work on that later, possibly trying to make some portable binaries.

Checked out the latest dataviewer into /opt/rtcds/caltech/c1/core/daq, however its not quite working yet either.  This is another thing Alex said he'll work on later.

We are also going to test Alex and Rolf's kernel patch over on c1iscex on Centos base kernel (apparently they've been using Gentoo up at hanford for the test stands...) and see how that works.

  3515   Thu Sep 2 16:45:48 2010 josephbUpdateCDSNumbering scheme of the PCI bus

Rolf has recently written a document describing how one should fill out an IO chassis and how the numbering works out.  This can be found in the DCC at Rolf's PCIe numbering guide (T1000523).

Basically it works out that slot 1 corresponds to PCIe number 1, but slot 2 corresponds to PCIe number 6.  And so forth.  The following table gives a quick summary.

Slot 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
PCIe Number 1 6 5 4 9 8 7 3 2 14 13 12 17 16 15 11 10


ELOG V3.1.3-