40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 40 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  4499   Thu Apr 7 13:14:23 2011 josephbUpdateCDSProposed plan for ITMX/ITMY control switch

Problem:

The controls (fast and slow both) think ITMX is ITMY and ITMY is ITMX.

Solution:

After some poking around today, I have convinced myself it is sufficient to simply swap all instances of ITMX for ITMY in the C1_SUS-AUX1_ITMX.db  file, and then rename it to C1_SUS-AUX1_ITMY.db (after having moved the original C1_SUS-AUX1_ITMY.db to a temporary holding file).

A similar process is then applied to the original C1_SUS-AUX1_ITMY.db file.  These files live in /cvs/cds/caltech/target/c1susaux.  This will fix all the slow controls.

To fix the fast controls, we'll modify the c1sus.mdl file located in /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/simLink/ so that the ITMX suspension name is changed to ITMY and vice versa.  We'll also need to clean up some of the labeling

At Kiwamu and Bryan's request, this will either be done tomorrow morning or on Monday.

So the steps in order are:

1) cd /cvs/cds/caltech/target/c1susaux

2) mv C1_SUS-AUX1_ITMX.db C1_SUS-AUX1_ITMX.db.20110408

3) mv C1_SUS-AUX1_ITMY.db C1_SUS-AUX1_ITMY.db.20110408

4) sed 's/ITMX/ITMY/g' C1_SUS-AUX1_ITMX.db.20110408 > C1_SUS-AUX1_ITMY.db

5) sed 's/ITMY/ITMX/g' C1_SUS-AUX1_ITMY.db.20110408 > C1_SUS-AUX1_ITMX.db

6) models

7) matlab

8) Modify c1sus model to swap ITMX and ITMY names while preserving wiring from ADCs/DACs/BO to and from those blocks.

9) code; make c1sus; make install-c1sus

10) Disable all watchdogs

11) Restart the c1susaux computer and the c1sus computer

 

  4509   Mon Apr 11 13:30:04 2011 josephb, JamieUpdateCDSNo Wiper script - Frames full over weekend

Problem:

The daqd process was dying every minute or so when it couldn't write frame.  This was slowing down the network by writing a 2.9G core dump over NFS every minute or so. (In /opt/rtcds/caltech/c1/target/fb/).

The problem was /frames/ was 100% full.

Apparently, when we switched the fb over to Gentoo, we forgot to install crontab and a wiper script.

Solution:

We will install crontab and get the wiper script installed.

  4510   Mon Apr 11 14:17:22 2011 josephb, jamieUpdateCDSFrame wiper script installed

[Joe, Jamie, Alex]

Fixes:

I asked Alex which cron to use (dcron? frcron?).  He promptly did the following:

emerge dcron

rc-update add dcron default

Copied the wiper.pl script from LLO to /opt/rtcds/caltech/c1/target/fb/

At that point, I modified wiper.pl script to reduce to 95% instead of 99.7%.

I added controls to the cron group on fb:

sudo gpasswd -a controls cron

I then added the wiper.pl to the crontab as the following line using crontab -e.

0 6 * * *       /opt/rtcds/caltech/c1/target/fb/wiper.pl --delete &> /opt/rtcds/caltech/c1/target/fb/wiper.log

Notes:

Note, placing backups on the /frames raid array will break this script, because it compares the amount in the /frames/full/, /frames/trends/minutes, and /frames/trends/seconds to the total capacity. 

Apparently, we had backups from September 27th, 2010 and March 22nd, 2011.  These would have broken the script in any case. 

We are currently removing these backups, as they are redundant data, and we have rsync'd backups of the frames and trends.  We should now have approximately twice the lookback of full frames.

  4518   Wed Apr 13 11:34:07 2011 josephbUpdateCDSFixed IFO_ALIGN.adl

Problem:

I switched the ITMX and ITMY control channels yesterday, but forgot to update the IFO_ALIGN.adl file (/opt/rtcds/caltech/c1/medm/c1ifo/) which had the control labels swapped to make life easier.

Solution:

I swapped ITMX and ITMY control locations on the screen.

Question:

Are there any other screens involving ITMX and ITMY that had controls reversed to make life easier?

  4522   Thu Apr 14 00:21:28 2011 KojiUpdateCDSNew C1LSC code running

[Jamie, Jenne, Koji]

We installed the new c1lsc and started the process.

We still need to configure bunch of the EPICS variables, matrices, and some of the filters.
This should be done in order to transmit the signals to the suspensions.
Jenne is going to work on this task tomorrow (Friday) morning,
and Koji will take over the task afternoon/evening.

  4524   Thu Apr 14 12:57:15 2011 josephbUpdateCDSRFM network happy again

[Joe, Alex]

Problem Symptoms:

There were red lights on the status screen indicating RFM errors for the c1scy, c1mcs and c1rfm processes.

The c1iscey, c1sus machines were receiving data sent over the RFM network from the c1ioo computer with a bad time stamp, a few cycles too late.  The c1iscex computer was receiving data from c1ioo fine.

Problem:

The c1iscex RFM card had gotten into a bad state and was somehow slowing things down/corrupting data.  It didn't affect itself, but due to the loop topology was messing everyone else up.  Basically the only one who wasn't throwing an error was the culprit.

Solution:

Hard power cycling the c1iscex computer reset the RFM card and fixed the problem.

  4543   Tue Apr 19 15:48:43 2011 josephbUpdateCDSMEDM screens and Front Ends updated to new Matrices

Problem:

The original matrix naming conventions for the front ends was broken.  It used _11, _12,...,_1e, _1f, _110, _111 and so forth.  The code was changes to use _1_1, _1_2,...,_1_16,_1_17, and so on.

In addition the matrix of filter banks was modified to use the same naming convetion (instead of starting at zero, it now start with one).

Work Done:

I rebuilt all the models, and restarted them all.

I wrote a simple script to modify the burt restore files to have the correct names for all the stored matrix values.

I also modified all the suspension screens, by modifying the default screens in /opt/rtcds/caltech/c1/medm/master/

The C1SUS, C1SCX, C1SPX, C1SCY, C1SPY, and C1MCS models had their foton filter files modified to put filters into the newly changed named filters

  4544   Tue Apr 19 17:34:02 2011 KojiUpdateCDSMEDM screens and Front Ends updated to new Matrices

Just a curiosity:

I just wonder how you have distingushed the difference between _111 and _111.

They are equivalent alone themselves. Have you looked at the contexts of the lines?
Or you just did not have the larger matrix than 16x16, did you?

  4545   Wed Apr 20 11:02:18 2011 josephbUpdateCDSMEDM screens and Front Ends updated to new Matrices
We simply didn't any matrices larger than 16x16. If we had, than that matrix would not have worked properly since the beginning.

Quote:

Just a curiosity:

I just wonder how you have distingushed the difference between _111 and _111.

They are equivalent alone themselves. Have you looked at the contexts of the lines?
Or you just did not have the larger matrix than 16x16, did you?

 

  4561   Fri Apr 22 12:07:38 2011 josephb, steveUpdateCDSRemoved hanging D-sub to SCSI in 1X2

Problem:

Way back, Jay had D-sub to SCSI adapters made to adapt our existing Sander box AA filters to the new SCSI based IO chassis.  However, these did not fit inside the box.

At the time, we simply left the cards outside hanging, which was a hack and needed to be replaced.

Solution:

Steve modified a black AA filter box so that it could fit the D-sub to SCSI adapter board on it, plus strain relief the SCSI cable, rather than let it hang.  The back of the box was cut, and an extending piece of metal attached to the bottom of the box.  The adapter board was screwed into the box, the SCSI plugged in, then the SCSI cable is clamped to the extending metal as well.

This modification will be propagated to the 3 remaining AA filter boards using the D-sub to SCSI adapter.

  4580   Thu Apr 28 10:53:50 2011 josephbUpdateCDSAdventures in Hyper-threading

What was done:

1) Turn off MC1, MC2, MC3, BS, ITMX, ITMY, PRM, SRM watchdogs.

2) Turn c1sus computer off (sudo shutdown now)

3) Go connect monitor and keyboard to c1sus.  Turn c1sus on.

4) Hit "del" key at the right time to go to setup (BIOS).

5) Go to BIOS advanced tab, CPU options, enable Multi-threading.

6) Hit F10 to save and let the computer continue booting.

What went wrong:

Once c1sus was up, I noticed several red lights and dead keep alives for the c1sus models.

Typing dmesg on c1sus revealed many messages like:

[  107.583420] c1x02: cycle 33737 time 20; adcWait 10; write1 0; write2 0; longest write2 0
[  107.583771] c1x02: cycle 33760 time 19; adcWait 11; write1 0; write2 0; longest write2 0

This indicates the Input/Output Processor (IOP) is not completing its duties within the 15 microseconds (1/64 kHz) that it has.  These lines indicate its take 20 or 19 microseconds.  (I saw messages ranging from 16 to 22 microseconds).

So this seems to agree with Rolf's observations that hyperthreading can cause a 5-10 microsecond increase in computation time.

So the next thing to do is modify which core the codes are running on, and try to get them paired up on the same physical core.

  4581   Thu Apr 28 12:25:11 2011 josephbUpdateCDSFurther adventures in Hyper-threading

First, I disabled front end starts on boot up, and brought c1sus up.  I rebuilt the models for the c1sus computer so they had a new specific_cpu numbers, making the assumption that 0-1 were one real core, 2-3 were another, etc.

Then I ran the startc1SYS scripts one by one to bring up the models.  Upon just loading the c1x02 on "core 2" (the IOP), I saw it fluctuate from about 5 to 12.  After bringing up c1sus on "core 3", I saw the IOP settle down to about 7 consistently.  Prior to hyper-threading it was generally 5. 

Unfortunately, the c1sus model was between 60 and 70 microseconds, and was producing error messages a few times a second

[ 1052.876368] c1sus: cycle 14432 time 65; adcWait 0; write1 0; write2 0; longest write2 0
[ 1052.936698] c1sus: cycle 15421 time 74; adcWait 0; write1 0; write2 0; longest write2 0

Bringing up the rest of the models (c1mcs on 4, c1rfm on 5, and c1pem on 6), saw c1mcs occasionally jumping above the 60 microsecond line, perhaps once a minute.   It was generally hovering around 45 microseconds.  Prior to hyper-threading it was around 25-28 microseconds.

c1rfm was rock solid at 38, which it was prior to hyper-threading.  This is most likely due to the fact it has almost no calculation and only RFM reads slowing it down.

c1pem continued to use negligible time, 3 microseconds out of its 480.

I tried moving c1sus to core 8 from core 3, which seemed to bring it to the 58 to 65 microsecond range, with long cycles every few seconds.

 

I built 5 dummy models (dua on 7, dub on 9, duc on 10, dud on 11, due on 1) to ensure that each virtual core had a model on it, to see if it helped with stabilizing things.  The models were basically copies of the c1pem model.

Interestingly, c1mcs seemed to get somewhat better and only taking to 30-32 microseconds, although still not as good as its pre-hyper-threading 25-28.  Over the course of several minutes it was no longer having a long cycle.

c1sus got worse again, and was running long cycles 4-5 times a second.

 

At this point, without surgery on which models are controlling which optics (i.e. splitting the c1sus model up) I am not able to have hyper-threading on and have things working.  I am proceeding to revert the control models and c1sus computer to the hyper-threading state.

 

 

  4583   Thu Apr 28 16:12:19 2011 josephb, jamieUpdateCDSNew CDS model SVN

New SVN

We are now using the LIGO CDS SVN for storing our control models.

The SVN is at:

https://redoubt.ligo-wa.caltech.edu/websvn/

The models are under cds_user_apps, then trunk, then approriate subsystem (ISC for c1lsc for example), c1 (for caltech 40m), then models.

We have checked out /cds_user_apps to /opt/rtcds/.

So to find the c1lsc.mdl model, you would go to /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsc.mdl

This SVN is shared by many people LIGO, so please follow good SVN practice.  Remember to update models ("svn update") before doing commits.  Also, after making changes please do an update to the SVN so we have a record of the changes.

New Practices

We are creating soft links in the /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/simLink/ to the models that you need to build.  So if you want to add a new model, please add it to the cds_users_apps SVN in the correct place and create a soft link to the simLink directory.

lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1sus.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1sus.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1sup.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1sup.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1spy.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1spy.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1spx.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1spx.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1scy.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1scy.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1scx.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1scx.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1mcs.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1mcs.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x05.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x05.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x04.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x04.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x03.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x03.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x02.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x02.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x01.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x01.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1rfm.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1rfm.mdl
lrwxrwxrwx 1 controls controls      55 Apr 28 14:41 c1dafi.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1dafi.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1pem.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1pem.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1mcp.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1mcp.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1lsp.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsp.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1lsc.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsc.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1ioo.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1ioo.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gpv.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gpv.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gfd.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gfd.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gcv.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gcv.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1ass.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1ass.mdl

  4590   Fri Apr 29 14:36:36 2011 josephb, steveUpdateCDSRemoved hanging D-sub to SCSI in 1X5

Quote:

Problem:

Way back, Jay had D-sub to SCSI adapters made to adapt our existing Sander box AA filters to the new SCSI based IO chassis.  However, these did not fit inside the box.

At the time, we simply left the cards outside hanging, which was a hack and needed to be replaced.

Solution:

Steve modified a black AA filter box so that it could fit the D-sub to SCSI adapter board on it, plus strain relief the SCSI cable, rather than let it hang.  The back of the box was cut, and an extending piece of metal attached to the bottom of the box.  The adapter board was screwed into the box, the SCSI plugged in, then the SCSI cable is clamped to the extending metal as well.

This modification will be propagated to the 3 remaining AA filter boards using the D-sub to SCSI adapter.

 The same modification was carried out at 1X5 for PRM & SRM.

Note:  D68L8EX-850Hz  are removed  and bypassed in 7 channels.

  4608   Tue May 3 10:41:35 2011 josephbUpdateCDSMorning maintenance

1) Filled in the C1SUS_BS_OLMATRIX properly so as to make the BS oplev work for Steve.

2) Turned on the ITMX damping.  Apparently it had tripped this morning, possibly due to work in the lab area.

3) The ETMX FE controller (c1scx) had ADC timed out and died sometime around 8:30 am.  The c1x01 (the IOP on the ETMX computer) was also indicating a FB status error (mismatch in DAQ channels).

The reported error in dmesg on c1iscex was:

[1628690.250002] c1spx: ADC TIMEOUT 0 3541 21 3605
[1628690.250002] c1scx: ADC TIMEOUT 0 3541 21 3605

Just to be safe, I rebuilt the c1x01 and c1scx models, ran ./activateDAQ.py, and used the scripts killc1spx, killc1scx, and killc1x01.

I finally restarted the process with startc1x01, startc1scx, and startc1spx.  Everything is currently alive and indicating all green.

  4609   Tue May 3 10:59:31 2011 josephbUpdateCDS1Y2 binary output adapter board now powered

I temporarily turned off the power to the 1Y2 rack this morning while wiring in the binary output adapter board power (+/- 15V) into the cross connects.

The board is now powered and we can proceed to testing if can actually control the LSC whitening filters.

  4647   Thu May 5 18:38:01 2011 ranaSummaryCDSSub-system TRAMP adjustments

I think that the gain ramping time (_TRAMP) should be set to 1 second for all filter modules by default. We don't want them to switch instantaneously except in a few special cases.

So Jamie and I wrote a script (in scripts/general/) which sets all of these fields to 1 for a given system. The name of the system is an argument to the script. e.g.

>  setTRAMP LSC 1

The idea is that we set it once and then from then on, its captured by the autoBURT. Of course, we have to run this script each time we add new filter modules to a model.

  4667   Mon May 9 16:12:49 2011 JamieConfigurationCDScanonicalize paths to core and userapps

I have updated the /opt/rtcds paths to reflect the new specification of the CDS aLIGO code release procedures document.


Path to RTS/opt/rtcds/caltech/c1/core/release

This is where the advLigoRTS front-end code generator is checked out.  The "release" directory is a link to the svn branch from which we are currently running ("trunk" by default).

This used to be at /opt/rtcds/caltech/c1/core/advLigoRTS


Path to userapps: /opt/rtcds/caltech/c1/userapps/release

This is where the cds_user_apps code is checked out.  cds_user_apps is where all of the front-end models, medm screen, scripts, etc. will live.  The "release" directory is a link to the svn branch from which we are currently running ("trunk" by default).

This was most recently at /opt/rtcds/cds_user_apps

 

  4668   Mon May 9 16:58:23 2011 JamieBureaucracyCDS!!!!REMEMBER TO COMMIT YOUR MODIFICATIONS!!!!

Whenever you modify any of the front-end code in /opt/rtds/caltech/c1/userapps, such as models, scripts or medm screens, REMEMBER TO COMMIT YOUR CHANGES, WITH A LOG MESSAGE!!!

cd /opt/rtcds/caltech/c1/userapps/trunk
svn commit path/to/modified/file

If you forget to commit things, they may be purged and you will loose your work.  ALWAYS COMMIT!

Things to note and watch out for:

  • When you commit, make sure you specify explicitly the path to the file that you are committing.  This repository is shared by all LIGO sites, including the sites (LHO and LLO) so you really want to make sure you're not committing anything that affects anything other than "c1" (ie. the 40m)
  • Only commit one file at a time, unless the changes to multiple files are tightly coupled.
  • Never commit without specifying the path to the file that you are committing.  Other people may be working on other files in the repository, and you don't want to commit their work with yours.
  • YOU MUST LEAVE A LOG MESSAGE.  Your log message should be specific to the file you have modified, and should be clear and verbose to explain exactly what you have done.
  • Use the "svn status" command to give you an overview of what is going on in the working directory.  This will tell you which files have been modified.  See "svn help status" for more information.

If you have questions ASK.  Don't force things.

  4669   Mon May 9 17:21:13 2011 JamieBureaucracyCDS!!!!REMEMBER TO COMMIT YOUR MODIFICATIONS!!!!

Also, remember to update the svn working copy before you begin doing any work, to make sure you're working on the most recent version:

cd /opt/rtcds/caltech/c1/userapps/trunk
svn update
  4679   Tue May 10 16:42:49 2011 josephbUpdateCDSNew upconversion model (c1uct)

[Ryan, Joe]

Ryan added the c1uct (upconversion tester) model to the c1ioo machine.   It is DCU_ID 32, CPU 6.  The relevant wiki page has been updated. It has been added to /diskless/root/etc/rtsystab on the fb machine and automatically comes up when the c1ioo computer is turned on. 

Joe still needs to add its status to the status screen.

It is soft linked from:

/opt/rtcds/caltech/c1/userapps/trunk/CDS/c1/models/c1uct.mdl

Ryan will expand upon its uses later.

  4680   Tue May 10 16:45:19 2011 josephbUpdateCDSc1ass now receiving AS55I from c1lsc

[Valera, Joe]

We added a cdsPCIx_SHMEM connection between the c1lsc and c1ass models.  This connection is called C1:LSC-ASS_AS55I, and sends the normalized AS55I data to Lockin 11 of the c1ass model.

In addition, in order to get the c1ass model to compile, we had to place all the non-IO parts inside a subsystem block, which we called ASS, and gave the top_names tag to.

The c1lsc and c1ass models were rebuilt, the frame builder restarted, and the models restarted.

  4706   Thu May 12 23:12:40 2011 kiwamuUpdateCDSc1lsc crashed

This is my third time to crash a real-time machine. This time I crashed c1lsc.

I physically rebooted c1lsc machine by pushing the power button and it came back and now running fine.

 

(what I did)

The story is almost the same as the last two times (1st time, 2nd time).

I edited c1lsc.ini file using daqconfig and then shutdown daqd running fb.

Some indicators for c1lsc on the C1_FE_STATUS screen became red. So I hit the 'DAQ reload' button on the C1LSC_GDS_TP screen.

Then c1lsc died and didn't respond to ping.

  4707   Thu May 12 23:41:47 2011 ranaUpdateCDSMEDM Snapshots not working

Looks like 2 different MEDM Snapshot functiions (at least) are broken.

The regular update of the screens here as well as the usual "Update Snapshot" and view "previous snapshot" button on all of the auto-generated screens.

Also, how do we add the snapshot button to the custom made screens?

  4718   Sun May 15 03:58:19 2011 ranaUpdateCDSdiagonalization of MC input matrix

There has been some input matrix diagonalization in the past by Yuta and Kiwamu, but I find the automation to be not totally satisfactory.

It would be better if we could automatically fit the data to find the Suspended optic eigenfrequencies and then use that to get the matrix. So I wrote a peak fitter to get the matrix.

It gets the data from mafalda with NDS2, then it makes the PSDs, and then starts with some initial guesses (based on looking at the plots) and them uses fminsearch to get the peak frequencies and Q's.

Using the output of this, we can use Yuta's method and take the passive transfer functions with the free swing data (from April 30, so we got do do it quick) to get the input matrix.

 

Doing the SUS input matrix is nice for having good damping (as long as we remember to include SIDE), but my motivation is to produce a good null stream from the 4 face sensors so that we can estimate the sensor noises at all times.

  4730   Tue May 17 11:45:20 2011 JamieConfigurationCDSpurged non-c1 site files from rtcds checkout of cds_user_apps
I purged all of the working copy checkouts of site files for all sites that are not c1 from the rtcds cds_user_apps working directory (/opt/rtcds/caltech/c1/userapps/trunk). I first checked that there were no outstanding changes, and then did the following (in bash):
cd /opt/rtcds/caltech/c1/userapps
rm -rf trunk
svn update --depth=files trunk
svn update --depth=empty trunk/{CDS,ISI,ISC,PSL,SUS}
svn update trunk/{CDS,ISI,ISC,PSL,SUS}/{c1,common}
  4732   Tue May 17 17:01:22 2011 jamieConfigurationCDSUpdate LSC channels from _DAQ to _DQ

As of RCG version 2.1, recorded channels use the suffix "_DQ", instead of "_DAQ".  I just rebuilt and installed the c1lsc model, which changed the channel names, therefore hosing the old daq channel ini file.  Here's what I did, and how I fixed it:

$ ssh c1lsc
$ cd /opt/rtcds/caltech/c1/core/trunk
$ make c1lsc
$ make install-c1lsc
$ cd /opt/rtcds/caltech/c1/scripts
$ ./startc1lsc
$ cd /opt/rtcds/caltech/c1/chans/daq
$ cat archive/C1LSC_110517_152411.ini | sed "s/_DAQ/_DQ/g" >C1LSC.ini
$ telnet fb 8087
daqd> shutdown

 

  4733   Tue May 17 18:09:13 2011 Jamie, KiwamuConfigurationCDSc1sus and c1auxey crashed, rebooted

c1sus and c1auxey crashed, required hard reboot

For some reason, we found that c1sus and c1auxey were completely unresponsive.  We went out and gave them a hard reset, which brought them back up with no problems.

This appears to be related to a very similar problem report by Kiwamu just a couple of days ago, where c1lsc crashed after editing the C1LSC.ini and restarting the daqd process, which is exactly what I just did (see my previous log).  What could be causing this?

  4746   Thu May 19 00:23:44 2011 ranaUpdateCDSdiagonalization of MC input matrix

 I've moved all of my SOS peak fitting stuff into the scripts area so that Leo can make it better:

/cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit

findPeaks.m gets the data and makes the fitted spectra that I put in the previous entry.

findMatrix.m is the barely started script that ought to take the TF data and output the matrix to the MEDM screen.

  4748   Thu May 19 12:09:41 2011 josephbUpdateCDSAA filter box pulled from 1X5, optic suspensions currently off

[Steve, Joe]

Steve pulled the top AA filter box from 1X5 which handled some of the suspensions channels.  We turned off all the watchdogs before pulling it out, as well as recorded which cables were connected to which inputs.

The case  is undergoing a structural modification to have the ADC adapter card which previously was loosely connected via cables, securely attached to the case.

Steve still wants to do some cabling in the rack while the box is out, and will return it this afternoon once he has finished that.

  4750   Thu May 19 17:53:03 2011 steveUpdateCDSAA filter box modified at 1X5

Quote:

[Steve, Joe]

Steve pulled the top AA filter box from 1X5 which handled some of the suspensions channels.  We turned off all the watchdogs before pulling it out, as well as recorded which cables were connected to which inputs.

The case  is undergoing a structural modification to have the ADC adapter card which previously was loosely connected via cables, securely attached to the case.

Steve still wants to do some cabling in the rack while the box is out, and will return it this afternoon once he has finished that.

 Job is done. Sus damping are back on. Cabling-strain reliefing are  not finished yet at 1X5 and 1X4

  4754   Fri May 20 03:29:04 2011 kiwamuUpdateCDSBinary IO box on 1X5 : LEDs off

[Steve / Kiwamu]

 When Steve was working on the strain reliefs on 1X5 he found that some LEDs on the back side of the binary IO boxes were off.

There are 4 binary IO boxes and their power are directly supplied from Sorensens. According to the display of the Sorensens, the power are correctly generated.

Steve and I checked a picture of the boxes taken before he started working and we found it's been like this.

It might be just a problem of the LEDs or the fuses are blown, but anyway it needs an inspection.

Here is a picture of the back side of the boxes. You can see some LEDs are on and some are off.

DSC_3050_small.jpg

  4764   Wed May 25 19:03:59 2011 JamieConfigurationCDSUpdate rtcds checkout of cds_user_apps with new top-level directory names.

The top-level subsystem subdirectories in the cds_user_apps source repository were renamed today to be all lower case.  This required checking out the new directory and updating all of the model links in /opt/rtcds/caltech/c1.  Here is how I updated the cds_user_apps working tree:

cd /opt/rtcds/caltech/c1/userapps
mv trunk{,.bak}
svn update --depth=files trunk
svn update --depth=empty trunk/{cds,isi,isc,psl,sus}

I then fixed the links in the /opt/rtcds/caltech/c1/core/release/src/epics/simLink directory:

for link in $(find . -maxdepth 1 -type l); do ln -sf $(readlink $link | tr [:upper:] [:lower:]) ; done

A couple of things had to be cleaned up:

  • /opt/rtcds/caltech/c1/userapps/trunk/cds/c1/models/c1uct.mdl was linked in, but that model doesn't seem to exist anymore, so I removed the link.
  • a couple of things were linked from /opt/rtcds/caltech/c1/userapps/trunk instead of /opt/rtcds/caltech/c1/userapps/release, so I fixed those links.
  • /opt/rtcds/caltech/c1/userapps/release/cds/test/models/llo/l1isctest.mdl was not checked out, so I checked it out and fixed the link (this model should really be named something different if it is of common use, or we plan on using it at the 40m).

 

  4765   Wed May 25 19:19:11 2011 JamieConfigurationCDS!!!CHECK IN YOUR MODELS!!!

!!!CHECK IN MODEL CHANGES!!!

Today I found three models that were modified, but not checked in to the SVN repository:

M       sus/c1/models/lib/sus_single.mdl
M       isc/c1/models/c1lsc.mdl
M       isc/c1/models/c1mcp.mdl

I checked in the c1lsc model, since I think it was just the change that Kiwamu made in http://nodus.ligo.caltech.edu:8080/40m/4749.  I left the others, since I have no idea what they are or who made them.

Please please please remember to commit your model changes to the SVN after you're done.  This is particularly important for important models, such as c1lsc.  If you don't check in your changes I can pretty much guarantee that at some point you will loose them.

 

  4770   Tue May 31 11:26:29 2011 josephbUpdateCDSCDS Maintenance

1) Checked in the changes I had made to the c1mcp.mdl model just before leaving for Elba.

2) The c1x01 and c1scx kernel modules had stopped running due to an ADC timeout. 

According to dmesg on c1iscex, they died at 3426838 seconds after starting (which corresponds to ~39 days).  "uptime" indicates c1iscex was up for 46 days, 23 hours. So my guess is about 8 days ago (last Monday or Tuesday),  they both died when the ADCs failed to respond quick enough for an unknown reason.

I used the kill scripts (in /opt/rtcds/caltech/c1/scripts/) to kill c1spx, c1scx, and c1x01.  I then used the start scripts to start c1x01, then c1scx, and then finally c1spx.  They all came up fine.

Status screen is now all green.  I renabled damping on ETMX and it seems to be happy. A small kick of the optic shows the approriately damped response.

  4772   Tue May 31 14:29:00 2011 jzweizigUpdateCDSframes

There seems to be something strange going on with the 40m frame builder.
Specifically, there is a gap in the frames in /frames/full near the start of
each 100k second subdirectory. For example, frames for the following times are missing:

990200042-990200669
990300045-990300492
990400044-990400800
990500032-990500635
990600044-990600725
990700037-990700704
990800032-990800677
990900037-990900719


To summarize, after writing the first two frames in a data directory, the next ~10 minutes of frames are usually missing. To make matters worse (for
the nds2 frame finder, at least) the first frame after the gap (and all successive frames) start at an arbitrary time, usually not aligned to a 16-second boundary. Is there something about the change of directories that is causing the frame builder to crash? Or is the platform/cache disk too slow to complete the directory switch-over without loss of data?

  4773   Tue May 31 15:45:37 2011 JamieUpdateCDSc1iscey IOchassis powered off for some reason. repowered.

We found that both of the c1iscey models (c1x05 and c1scy) were unresponsive, and weren't coming back up even after reboot.  We then found that the c1iscey IOchassis was actually powered off.  Steve's accepts some sort of responsibility, since he was monkeying around down there for some reason.  After powerup and reboot, everything is running again.

  4776   Wed Jun 1 11:31:50 2011 josephbUpdateCDSMC1 LR digital reading close to zero, readback ~0.7 volts

There appears to be a bad cable connection somewhere on the LR sensor path for the MC1 optic.

The channel C1:SUS-MC1_LRPDMon is reading back 0.664 volts, but the digital sensor channel, C1:SUS-MC1_LRSEN_INMON, is reading about -16.  This should be closer to +1000 or so.

We've temporarily turned off the LRSEN filter module output while this is being looked into.

I briefly went out and checked the cables around the whitening and AA boards for the suspension sensors, but even after wiggling and making sure everything was plugged in solidly.  There was one semi-loose connection, but it wasn't on the MC1 board, but I pushed it all the way in anyways.  The monitor point on the AA board looks correct for the LR channels, although ITMX LR struck me as being very low at about -0.05 Volts.

According to data viewer, the MC1 LR sensor channel went bad roughly two weeks ago, around 00:40 on 5/18 UTC, or 17:40 on 5/17 PDT.

 

UPDATE:

It appears the AA board (or possibly the SCSI cable connected to it) is the problem in the chain.

  4781   Thu Jun 2 16:31:41 2011 JamieUpdateCDSaquired SUS channel name suffixes changed from _DAQ to _DQ

CDS changed the suffix for all aquired channel names from _DAQ to _DQ.  When we rebuilt the sus models, described in the previous log, the channel names were changed and consequently the channel files were completely rewritten.

To fix the issue, the latest archived channel file was copied back into the chans directory, and the suffixes were changed, as so:

cd /opt/rtcds/caltech/c1/chans
cp archive/C1SUS_110602_155403.ini  C1SUS.ini
sed -i 's/DAQ/DQ/g' C1SUS.ini

We then restarted the models and the framebuilder.

  4790   Mon Jun 6 18:29:01 2011 Jamie, JoeUpdateCDSCOMPLETE FRONT-END REBUILD (WITH PROBLEMS (fixed))

Today Joe and I undertook a FULL rebuild of all front end systems with the head of the 2.1 branch of the RCG.  Here is the full report of what we did:

  1. checked out advLigoRTS/branches/branch-2.1, r2457 into core/branches/branch-2.1
  2. linked core/release to branches/branch-2.1
  3. linked in models to core/release/src/epics/simLink using Joe's new script (userapps/release/cds/c1/scripts/link_userapps)
  4. remove unused/non-up-to-date models:
  5. c1dafi.md
    c1lsp.mdl
    c1gpv.mdl
    c1sup_vertex_plant_shmem.mdl
  6. modified core/release/Makefile so that it can find models:
  7. --- Makefile	(revision 2451)
    +++ Makefile (working copy)
    @@ -346,7 +346,7 @@
    #MDL_MODELS = x1cdst1 x1isiham x1isiitmx x1iss x1lsc x1omc1 x1psl x1susetmx x1susetmy x1susitmx x1susitmy x1susquad1 x1susquad2 x1susquad3 x1susquad4 x1x12 x1x13 x1x14 x1x15 x1x16 x1x20 x1x21 x1x22 x1x23

    #MDL_MODELS = $(wildcard src/epics/simLink/l1*.mdl)
    -MDL_MODELS = $(shell cd src/epics/simLink; ls m1*.mdl | sed 's/.mdl//')
    +MDL_MODELS = $(shell cd src/epics/simLink; ls c1*.mdl | sed 's/.mdl//')

    World: $(MDL_MODELS)
    showWorld:
  8. removed channel files for models that we know will be renumbered
    • For this rebuild, we are also building modified sus models, that are now using libraries, so the channel numbering is changing.
  9. make World
    • this makes all the models
  10. make installWorld
    • this installs all the models
  11. Run activateDQ.py script to activate all the relevant channels
    • this script was modified to handle the new "_DQ" channels
  12. make/install new awgtpman:
  13. cd src/gds
    make
    cp awgtpman /opt/rtcds/caltech/c1/target/gds/bin
  14. turn off all watchdogs
  15. test restart one front end: c1iscex
  16. BIG PROBLEM

    The c1iscex models (c1x01 and c1scx) did not come back up.  c1x01 was running long on every cycle, until the model crashed and brought down the computer.  After many hours, and with Alex's help, we managed to track down the issue to a patch from Rolf at r2361.  The code included in that patch should have been wrapped in an "#ifndef RFM_DIRECT_READ".  This was fixed and committed to branches/branch-2.1 at r2460 and to trunk at r2461.

  17. update to core/branches/branch-2.1 to r2460
  18. make World && make installWorld with the new fixed code
  19. restarted all computers
  20. restart frame builder
  21. burt restored to 8am this morning
  22. turned on all watchdogs

Everything is now green, and things seem to be working.  Mode cleaner is locked.  X arm locked.

 

  4800   Thu Jun 9 16:18:03 2011 josephbUpdateCDSSecond trends only go back 12 days

While answering a quick question by Kiwamu, I noticed we only had second trends going back to 99050000 GPS time, May 27th 2011. 

Trends (I thought) were intended to be kept forever, and certainly longer than full data, which currently goes back several months.

Jamie will need to look into this.

  4801   Thu Jun 9 18:25:22 2011 kiwamuHowToCDSlook back a channel which doesn' exist any more

For some purposes I looked back the data of some channels which don't exist any more.  Here I explain how to do it.

If this method is not listed on the wiki, I will put this instruction on a wiki page.

 

(How to)

   (1) Edit an "ini" file which is not associated to the real-time control (e.g. IOP_SLOW.ini)

   (2) In the file, write a channel name which you are interested in. The channel name should be bracketed like the other existing channels.

               example:  [C1:LSC-REFL11_I_OUT_DAQ]

   (3) Define the data rate. If you want to look at the full data, write

              datarate = 2048

        just blow each channel name.

        Or if you want to look at only the trends, don't write anything.

   (4) Save the ini file and restart fb. If necessary hit "DAQ Reload" button on a C1:AAA_GDS_TP.adl screen to make the indicators green.

   (5) Now you should be able to look at the data for example by dataviewer.

   (6) After you finish the job, don't forget to clean up the sentences that you put in the ini file because it will always show up on the channel list on dtt and is just confusing.

        Also don't forget to restart fb to reflect the change.

  4803   Fri Jun 10 12:02:10 2011 ranaUpdateCDSSecond trends only go back 12 days

Quote:

While answering a quick question by Kiwamu, I noticed we only had second trends going back to 99050000 GPS time, May 27th 2011. 

Trends (I thought) were intended to be kept forever, and certainly longer than full data, which currently goes back several months.

Jamie will need to look into this.

 Our concept is to keep second trends for 1-2 months and minute trends forever. The scheme that Alan had worked out many years ago had it such that we could look back to 1998 and that the minute trends would be backed up somehow.

If its not working, we need to get Alan's help to recover the previous configuration.

  4809   Mon Jun 13 15:33:55 2011 Jamie, JoeUpdateCDSDolphin fiber between 1Y3 and 1X4 appears to be dead

The fiber that connects the Dolphin card in the c1lsc machine (in the 1Y3 rack) to the Dolphin switch in the 1X4 rack appears to have died spontaneously this morning.  This was indicated by loss of Dolphin communication between c1lsc and c1sus.

We definitively tracked it down to the fiber by moving the c1lsc machine over to 1X4 and testing the connection with a short cable.  This worked fine.  Moving it back to using the fiber again failed.

Unfortunately, we have no replaced Dolphin fiber.  As a work around, we are stealing  a long computer->IO chassis cable from Downs and moving the c1lsc machine to 1X4.

This is will be a permanent reconfiguration.  The original plan was to have the c1lsc machine also live in 1X4.  The new setup will put the computer farther from the RF electronics, and more closely mimic the configuration at the sites, both of which are good things.

  4811   Mon Jun 13 18:40:08 2011 Jamie, JoeUpdateCDSSnags in the repair of LSC CDS

We've run into a problem with our attempts to get the LCS control back up and running.

As reported previously, the Dolphin fiber connection between c1lsc and c1sus appears to be dead.  Since we have no replacement fiber, the solution was to move the c1lsc machine in to the 1X4 rack, which would allow us to use one of the many available short Dolphin cables, and then use a long fiber PCIe extension cable to connect c1lsc to it's IO chassis.  However, the long PCIe extension cable we got from Downs does not appear to be working with our setup.  We tested the cable with c1sus, and it does not seem to work.

We've run out of options today.  Tomorrow we're going to head back to Downs to see if we can find a cable that at least works with the test-stand setup they have there.

  4812   Mon Jun 13 19:26:42 2011 Jamie, JoeConfigurationCDSSUS binary IO chassis 2 and 3 moved from 1X5 to 1X4

While preping 1X4 for installation of c1lsc, we removed some old VME crates that were no longer in use.  This freed up lots of space in 1X4.  We then moved the SUS binary IO chassis 2 and 3, which plug into the 1X4 cross-connect, from 1X5 into the newly freed space in 1X4.  This makes the cable run from these modules to the cross connect much cleaner.

  4815   Tue Jun 14 09:25:17 2011 JamieUpdateCDSDolphin fiber tested with c1sus, still bad

The bad Dolphin was still bad when tested with a connection between c1sus and the Dolphin swtich.

I'm headed over to Downs to see if we can resolve the issue with the PCIe extension fiber.

  4816   Tue Jun 14 12:23:44 2011 Jamie, JoeUpdateCDSWE ARE ALL GREEN! LSC back up and running in new configuration.

After moving the c1lsc computer to 1X4, then connecting c1lsc to it's IO chassis in 1Y3 by a fiber PCIe extension cable, everything is back up and running and the status screen is all green.  c1lsc is now directly connected to c1sus via a short copper Dolphin cable.

After lunch we will do some more extensive testing of the system to make sure everything is working as expected.

  4837   Mon Jun 20 09:28:19 2011 JamieUpdateCDSShutting down low-voltage DC power in 1X1/1X2 racks

In order to install the BO module in 1X2, I need to shut down all DC power to the 1X1 and 1X2 racks.

  4838   Mon Jun 20 10:45:43 2011 JamieUpdateCDSPower restored to 1X1/1X2 racks. IOO binary output module installed.

All power has been restored to the 1X1 and 1X2 racks.  The modecleaner is locked again.

I have also hooked up the binary output module in 1X2, which was never actually powered.  This controls the whitening filters for MC WFS.  Still needs to be tested.

ELOG V3.1.3-