Updated the c1scx model to have two Lockin demodulators (C1:SUS-ETMX_LOCKIN1 and C1:SUS-ETMX_LOCKIN2).  There is a matrix C1:SUS-ETMX_INMUX which directs signals to the inputs of LOCKIN1 and LOCKIN2.  Currently only the GREEN_TRX signal is the only signal going in to this matrix, the other 3 are grounds.  The actual clocks themselves had to be at the top level (they don't work inside blocks) and thus named C1:SCX-ETMX_LOCKIN1_OSC and C1:SCX-ETMX_LOCKIN2_OSC.

There is a signal (IPC name is C1:GCV-SCX_GREEN_TRX) going from the c1gcv model to the c1scx model, which will contain the output from Jenne's green transmission PD which will eventually be placed. I've placed a filter bank on it in the c1gcv model as a monitor point, and it corresponds to C1:GCV-GREEN_TRX.

The suspension control screens were modified to have a screen for the Matrix feeding signals into the two lockin demodulators.  The green medm screen was also modified to have readbacks for the GREEN_TRX and GREEN_TRY channels.

So on the board, the top channel (labeled 1, corresponds to code ADC_0_0) is MCL.

Channel 2 (ADC_0_1) is assigned to frequency divided green signal.

Channel 3 (ADC_0_2) is assigned to the beat PD's DC output.

Channel 4 (ADC_0_3) is assigned to the green power transmission for the x-arm.

Channel 5 (ADC_0_4) is assigned to the green power transmission for the y-arm.

[Joe,Alex]

Alex came over and we installed the new Dolphin drivers so that the front ends using the Dolphin PCIe RFM network don't pause for a long time when one of the other nodes in the network go down.  Generally this pause would cause the code to time out and quit.  Now you can take c1lsc or c1sus down without having the other have problems.

We did note on reboot however, that the Dolphin_wait script sometimes (not always) seems to hang.  Since this is run at boot up, to ensure the dolphin card has had enough to allocate memory space for data to be written/read from by the IOP process, it means nothing else in the startup script gets run if it does happen.  In this case, running "pkill dolphin_wait" may be necessary.

Note that you may still have problems if you hit the power button to force a shutdown (i.e. holding it for 4 seconds for immediate power off), but as long as you do a "reboot" or "shutdown -r now" type command, it should come down gracefully. 

What was done:

Alex grabbed the code from his server, and put it /home/controls/DIS/ on fb.

He ran the following commands in that directory to build the code.

./configure  '--with-adapter=DX' '--prefix=/opt/DIS'

make

sudo make install

He proceeded to modify the /diskless/root/etc/rc.local to have the line:

insmod /lib/modules/2.6.34.1/kernel/drivers/dis/dis_kosf.ko

In that same file he commented out

cd /root

and

exec /bin/bash/

He then modified the run levels in /diskless/root/etc/inittab. Level 0, level 3, and level 6 were changed:

l0:0:wait/etc/rc.halt

l3:3:wait:etc/rc.level3

l6:6:wait:/etc/rc.reboot

Then he created the scripts he was refering to:

rc.level3 is just:

exec /bin/bash

rc.halt is:

/opt/DIS/sbin/dxtool prepare-shutdown 0
sleep 3
halt -p

rc.reboot is:

reboot

Basically rc.halt calls a special code which prepares the Dolphin RFM card to shutdown nicely.  This is why just hitting the power button for 4 seconds will cause problems for the rest of the dolphin network.

We then checked out of svn the latest dolphin.c in  /opt/rtcds/caltech/c1/core/advLigoRTS/src/fe

The Dolphin RFM cards have a new numbering scheme.  4 is reserved for special  broadcasts to everyone, so the Dolphin node IDs now start at 8.  So we needed to change the c1lsc and c1sus Dolphin node IDs.

To change them we went to /etc/dis/dishosts.conf on the fb machine, and changed the following lines:

HOSTNAME: c1sus
ADAPTER:  c1sus_a0 4 0 4
HOSTNAME: c1lsc
ADAPTER:  c1lsc_a0 8 0 4

to

HOSTNAME: c1sus
ADAPTER:  c1sus_a0 8 0 4
HOSTNAME: c1lsc
ADAPTER:  c1lsc_a0 12 0 4

The FE models for the c1lsc and c1sus machines were recompiled and then the computers were rebooted.  After having them come back up, we tested that there was no time out by shutting down c1lsc and watching c1sus. We then reveresed and shutdown c1sus while watching c1lsc.  No problems occured.  Currently they are up and communicating fine.

This is just a listing of  CDS problems I still notice today:

1. MC2-MCL button was left ON due to BURT failure. This, of course, screws up our Green locking investigations because of the unintended feedback. Please fix the BURT/button issue.

2. The GCV - FB0 status is RED. I guess this means there's something wrong? Its really a bad idea to have a bunch of whited out or falsely red indicators. No one will ever use these or trust these in the future.
3. MC1/2/3 Lockins are all white. Also, the MODE switches for the dewhitening are all white.
4. Is the MC SIDE coil dewhitening filter synced with anything? It doesn't seem to switch anything. Maybe the dewhite indicators at the top right of the SUS screens can be made to show the state of the binary output instead of just the digital filter
5. MC WFS is all still broken. We need a volunteer to take this on - align beams, replace diodes, fix code/screens.

I propose to use C1:ALS-xxx_xxx for the names of the green related channels, instead of GCV, GCX, GCY, GFD...

Like C1:SUS or C1:LSC, we name the channels by the subsystems first, then probably we can specify the place.

We can keep the names of the processes as they are now.

  I am concentrating on the RF system just now and will be attending to the RF PDs one by one.  Also plan to work on some of the simpler CDS problems when I overlap with Joe.  Will be available for helping out with the beam alignment.

[Joe, Alex]

This morning I began the process of bringing our copy of the CDS code up to date to the version installed at Livingston. The motivation was to get fixes to various parts, among others such as the oscillator part.   This would mean cleaning up front end model .mdl files without having to pass clk, sin, cos channels for every optic through 3 layers of simulink boxes.

I also began the process of using a similar startup method, which involved creating /etc/init.d/ start and stop scripts for the various processes which get run on the front ends, including awgtpman and mx_streams.  This allows the monitor software called monit to remotely restart those processes or provide a web page with a real time status of those processes.  A cleaner rc.local file utilizing sub-scripts was also adapted.

I did some testing of the new codes on c1iscey.  This testing showed a problem with the timing part of the code, with cycles going very long.  We think it has something to do with the code not accounting for the fact that we do not have IRIG-B timing cards in the IO chassis providing GPS time, which the sites do have.  We rely on the computer clock and ntpd.

At the moment, we've reverted to svn revision 2174 of the CDS code, and I've put the previously working version of the c1scy and c1x05 (running on the c1iscey computer) back. Its from the /opt/rtcds/caltech/c1/target/c1x05/c1x05_11014_163146 directory.  I've put the old rc.local file back in /diskless/root/etc/ directory on the fb machine.  Currently running code on the other front end computers was not touched.

Although Joe and Kiwamu claim that they have inserted the correct DAQ names for the OPLEVs (e.g. PERROR and YERROR) back in Jan. 11, when I look today, I see that these channels are missing!

I want my PERROR/YERRORs back!

Currently, there is a test directory called /opt/rtcds/caltech/c1/new_core where we have the latest svn checkout.  Tomorrow (after everything works), it will become the core directory.

1) Modify on the fb machine the /diskless/root/etc/ld.so.cache file.  This is done by logging into fb, going to /etc/ld.so.conf.d/, modifying epics-x86_64.conf to only have .10 stuff , and running sudo /sbin/ldconfig.  Copy the newly generated /etc/ld.so.cache file to /diskless/root/etc/.

2) Modify the rc.local file on the fb machine in /diskless/root/etc/ to take advantage of the new subscripts and init.d/ start scripts.

3) Add the no_rfm_dma to all the iop models (c1x01,c1x02,c1x03,c1x04,c1x05).

4) Rebuild all front end models with new code.  Install.

5) Build awgtpman and mx_streams with new code.

6) Rerun activateDaq.py (to fix channel names from all the rebuilt code).

7) Double check Burt request files have the switch fix.

8) Restart the front ends.

9)Restart the frame builder.

9) Check channels, exitations, RFM connections.

10) Check Monit is working.

[Joe,Valera]

Valera pointed out the OPLEV SUM channels were incorrect.  We changing the optical level sum channel to _OPLEV_SUM when it should have been OL_SUM.  This has been fixed in the activateDAQ.py script.

1. ETMX cannot load his filter coefficients. Even if I change the file, the load button doesn't work. I tried changing the lockin filters but they won't load.
2. The ETMX filter modules appear to have 2048 for some of the modules and 16384 for some of the others. How come the make script doesn't make them all 16384?
3. There are a bunch of kill/start scripts in the scripts/ directory instead of in the scripts/FE/ directory. Did this get reverted after a new code exchange?
4. I tried restarting the code using c1startscx, but that doesn't work correctly. It cannot find the burtrb and burtwb files even though they are in the normal path.
5. Kiwamu was using a bunch of cockamamie filters I found.
6. I can't get any minute trend data. I tried on rossa, rosalba, and op440m.

I've modified the rc.local file to run the IOC codes as controls, which means they no longer write root permission log files on startup.

The awgtpman, which was the other permission issue with the start scripts, is started by a run script now.  This new version seems to be content to keep the permissions of the current log file, which is set to controls.

This should prevent the issue of sudo wiping your path environment variable for just that command. (Try "sudo which burtwb" versus "which burtwb" for example).  This apparently a security feature of sudo.

If you should happen to use sudo to run a start script, the easiest solution to fix the permissions is just got to the target directory (type "target") and run "sudo chown controls:controls -R *" on one of the workstations (the front ends don't handle the groups properly at the moment).

This should allow the scripts to properly use burtrb and burtwb to write and backup burt files.

I modified the core/advLigoRTS/Makefile to once again place the startc1SYS and killc1SYS scripts in the scripts/FE/ directory.

It had been reverted in the SVN update.

Problem:

Could not load filters into the C1:SUS-ETMX_LOCKIN1_SIG filter bank.

Reason:

Apparently the filter bank name was too long.  I'm not sure why this isn't caught by the real time code generator, I'm planning on asking Alex and Rolf about it today.

Solution:

Reduce the name of the components.  Basically LOCKIN1 needs to become something like LOCK1 or LIN1.

In related news, it looks like the initial filters are hard coded to be 2048 Hz.  Given that they start out empty they won't cause things to break immediately, and if you're editing the file you can update the rate as you add the filter.  I'll also bring this up with Alex and Rolf and see if the RCG can't be more intelligent about its filter generation.

As one of the trouble shooting steps for the daqd (i.e. framebuilder) I rebuilt the daqd executable.  My guess is somewhere in the build code is some kind of GPS offset to make the time correct due to our lack of IRIG-B signal.

The actual daqdrc file was left untouched when I did the new install, so the symmetricom gps offset is still the same, which confuses me.

I'll take a look at the SVN diffs tomorrow to see what changed in that code that could cause a 300000000 or so offset to the GPS time.

THANK YOU, JOE !!! 

I talked to Alex today and had two things fixed:

First the maximum length of filter names (in the foton C1SYS.txt files in /chans) has been increased to 40, from 20.  This does not increase EPICS channel name length (which is longer than 20 anyways).

This should prevent running into the case where the model doesn't complain when compiled, but we can't load filters.

Additionally, we modified the feCodeGen.pl script in /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/util/ to correctly generate names for filters in all cases.  There was a problem where the C1 was being left off the file name when in the simulink .mdl file the filter was located in a box which had "top_names"  set.

We updated the c1ass model to include the BS.  We removed the dither excitation of the PZTs.  PZT control goes to epics. To do this, modified the /cvs/cds/caltech/target/c1iscaux/PZT_AI.db file.  We basically have it sum both the existing epics slider and our new output from c1ass.

More importantly we updated the color scheme.

We compiled and tested the Dolphin and RFM which work.

I should note we can't figure out why testpoints are not working properly with just this model.  Alex and Joe spent well over an hour trying to debug it to no success.  Current workaround is to add what channels you want from c1ass to the DAQ recording.  Other testpoints on other models appear to be working.

Alex and I updated the open mx drivers from 1.3.3 to 1.3.901 (1.4 release candidate).  We downloaded the drivers from: http://open-mx.gforge.inria.fr/

We put them in /root/open-mx-1.3.901 on the fb machine (and thus get mounted by all the front ends.).  We did configure and make and make install.

We did a quick check with /opt/mx/bin/mx_info on the fb machine at this point and realized the MAC addresses and host names were all messed up, including two listings for c1iscex with different mac addresses (neither of which was c1iscex).

We then brought all the front ends mx_streams down, brought the fb down, then cleared all the peer names with mx_hostname.  We then brought the fb up, and the front end mx_stream processes.

/opt/mx/mx_info now shows a clean and correct set of hostnames and mac addresses.  Testpoints and trends are working.

[Joe, Jamie]

We modified the c1scx model to have a switch to go between simulated and real plants.  The channel is currently C1:SCX-SIM_SWITCH. 

When this channel is zero, the simulated plant channels are going to the ADCs and zeros  are going out to the real DACs.  When this channel is one, the real ADCs are coming in, and real data is going out to the DACs.

Jamie will be adding a big green/red light to the suspension screens which indicate the state of the simulated plant.  We will also eventually add this to the overall status screen.

A control screen for the simulated plant is located at /opt/rtcds/caltech/c1/medm/c1spx/master/C1SUP_ETMX.adl.  These are currently a work in progress.

[Joe/Kiwamu]

 We found there are some filter names that we can not properly build for some reason.

The following names are not properly going to be built :

 - REFL_DC

 - AUX

If we use the names shown above for filters, it doesn't compile any filter modules.

We took a quick look around the src files including feCodegen.pl, but didn't find any obvious bugs.

The screens for the simplified c1spx model have been updated.  I re-introduced the suspension point information into the sensor output matrix so we can take into account the fact that as the entire supporting structure moves, the osems moves relative to the optic.

Master screens for the noise filters (i.e. 60 Hz, suspension point motion, and optic noise) have been created.

I have currently set the matrix values of the c1spx model to handle just longitudinal motion.  I.e. Coils drive only in the POS degree of freedom and sensor read outs are also only in the POS degree of freedom.  I've turned off all the noise inputs.

I added a simple double pole at 1 Hz in the C1:SUP_ETMX_PL_F2P_0_0 filter bank.

For some reason the c1ioo machine suddenly died just 30 miteus before.

It died after we added a DAQ channel for c1gcv and rebooted the frame builder.

It didn't respond to a ping command. Therefore I rebooted the machine by clicking the physical reset button.

Now it seems fine.

Today was a banner day for Simulated Plants.

Joe and Jamie have been working to get it all happening and this afternoon we started stuffing filters into the plant to make it act like the:

We put in the following features so far:

1. Anti-Imaging filters (these are hacked to be approximate since the real ones are 7570 Hz LP filters and the SimAI only can have filters up to 8192 Hz).
2. Dewhitening filters (copied from the SimDW in the SUS-ETMY screens)
3. Coil Driver transimpedance (1 / 200 Ohms)
4. Magnet-coil force constant (0.016 N/A)
5. Conversion from Coil to DOF Basis
6. All DOFs of the mechanical model are represented as simple harmonic oscillators with Q~100 and f ~ measured free swinging peaks.
7. Signals/Noise can be injected either as force noise on the test mass or as displacement noise at the suspension point.
8. Conversion from DOF to Shadow Sensor basis.
9. Optical Levers (with whitening)
10. Shadow Sensors have 2V/mm readout gain and whitening filters before being digitized by the SimADC.

We have also changed the switching logic for the SUS and SimETMs for the shadow sensor whitening. It used to be that either the hardware OR the software whitening was on. Now we have made it like all of the other whitening/antiwhitening in LIGO and the whitening/antiwhitening come on together. Joe and Jamie are going to propagate this to the other SUS. The hardware filter is a 30,100:3 (poles:zeros) whitening filter. The digital filter used to also be 30,100:3 with a DC gain = 1. I've changed the FM1 filter in the XXSEN filter banks into a 3:30 for the ETMY so that it now comes on and just compensates the hardware filter. This change should be propagated to all other SUS and the MEDM screens updated to show the new situation.

After this change, we decided to calibrate the {UL,UR,LL,LR,SD}SEN channels into units of microns. To do this we have made an FM6 filter called 'cts2um' that accounts for the OSEM gain and the ADC conversion factors. These channels are now in units of microns without applying any calibration in the DTT or Dataviewer. The plot below shows the OSEM shadow sensor time series with all damping loops ON and a very rough version of seismic noise being injected in all 6 DOFs (note that the y-axis is microns and the x-axis is seconds).

Next, Jamie is adding the angular calibrations (so that SUSPIT and SUSYAW are in rads) and Chris is making vectift quality seismic noise injectors.

We also need to add coating thermal noise, suspension thermal noise, substrate thermal noise, ADC/DAC noise and a lot of MEDM screen indicators of what state we're in. I myself can't tell from the OSEM time series if its real or Sim.

To simulate digitization noise, the easiest way I found was to use the MathFunction block, found in the CDS_PARTS model, under simLinkParts. 

The MathFunction block supports square of input value, square root of input value, reciprocal of input value, and modulo of two input values.

The last is useful because it casts the input values as integers before taking the modulo.By placing this block after the saturation block (set to +/- 32768), adding 32768.5, choosing the 2nd input to be larger than 2 * 32768 (100,000 in this case), and then subtracting 32768, we wind up with a rounding function. 

The above method has been applied to the c1spy model in the CI and SO out sub-blocks.

While fixing up some medm screens and getting spectra of the simulated plant, I realized that the naming convention for the Matrices of Filter banks was backwards when compared to that of the normal matrices (and the rest of the world).  The naming was incorrectly column, row.

This has several ramifications:

1) I had to change the suspensions screens for the TO_COIL  output filters.

2) I had to change the filters for the suspension with regards to the TO_COIL output filters so they go in the correct filter banks.

3) Burt restores to times previous March 11th around noon, will put your TO_COIL output filters in a funny state that will need to be fixed.

4) The simplant RESPONSE filters had to be moved to the correct filter banks.

5) If you have some model I'm not aware of that uses the FiltMuxMatrix piece, it is going to correctly build now, but you're going to have to move filters you may have created with foton.

The FM1 filter module change for XXSEN was propagated to the ETMX suspension.  This was a change from a 30,100:3 with a DC gain of 1 to a 3:30 which just compensates the hardware filter.

The good gains for the Sim damping were found to be:  100 for ETMX_SUSPOS, 0.1 ETMX_SUSPIT, and 0.1 ETMX_SUSYAW, ETMX_SUSSIDE is -70.  Gains much higher tended to saturate the simulated coils (i.e. hitting 10V limit) and then had to have the histories cleared for the RESPONSE matrix.

These seem to work to damp the real ETMX as well.

I did some work on the ETMY real and Sim.

1. Set SUS coil gains to have the same quadropole arrangement as the magnets do (-1, 1, 1, -1) so that POS = POS instead of pringle.
2. Set the Sim Magnet polarities to match this. These are the ETMY_CI filter banks.
3. Found that the Xycom cable from the ETMY slow controls was unplugged at the Xycom side. This was preventing enabling the ETMY coil driver and so there was no real damping of the suspension going on. I plugged it in and checked that the mirror could now be moved.
4. The C1SUS_ETMY master screen's BLUE output filter area is now mis-labeled. If you trust the screen you would set it up to drive the suspension incorrectly. This MUST be fixed along with all of the other misleading features of the screen.
5. ETMY SUSSIDE filter bank had a 2048 Hz sample rate and was making the damping not work correctly. Fixed to 16384 Hz.
6. 12 Hz, 4th order Cheby low pass added and turned on for the local damping filtering. This is not optimum, but its just there to give us some filtering without introducing some instability via phase lag around 3 Hz.
7. ETMY OL beam re-aligned on ISCT-EX.
8. TM Offset buttons not working on the main overview screen.

It seems like there is still a problem with the input whitening filters. I believe the Xycom logic is set such that the analog whitening of the OSEM signals is turned ON only when the FM1 is turned OFF. Joe has got to fix this (and elog it) so that we can damp the suspension correctly. For now, the damping of the ETMY and the SETMY require different servo gains and signs, probably because of this.

4. The blue Output Filters  section has been changed to agree with the new filter of matrices row, column labeling.  My fault for not testing it and realizing it was broken.  The change was made in /opt/rtcds/caltech/c1/medm/master/C1SUS_DEFAULTNAME.adl and then ,/generate_master_screens.py was run, updating all the screens.

5.  I have swapped the logic for the sensor filter banks (ULSEN, URSEN, etc).  It now sends a "1" to the Binary Output board controlling the OSEM analog whitening when the FM1 filter is ON.  This has been done for all the suspensions (BS, ITMX,ITMY, SRM, PRM, MC1, MC2,MC3, ITMX, ITMY).

I am also updating the first sensor filter banks for the BS, ITMX, ITMY, SRM, PRM,MC1,MC2,MC3, called "3:30", to match the Y and X ends.

8.  I can't find any documentation on how to get a momentary button press to toggle states.  I could stick a filter bank in and use the on/off feature of that part, but that feels like a silly hack.  I've decided for the moment to split the TM offset button into 2, one for ON, one for OFF.  I'll put in on the list of things to have added to the RCG code (either a method, or documentation if it already exists). 

EDIT:  TM offset still doesn't work.  Will worry about it next week.

9.  Fixed a connection in SPY/SPX models where  the side senor path that was missing a constant to a modulo block.

[Joe, Alex]

Steve pointed out to me today he couldn't get trends for his PEM slow channels like C1:PEM-count_full. 

I experimented a bit and found for long time requests (over 20 days), it would produce minute trends up to the current time, but only if they started far enough back.  So the data was being written, but something was causing a problem for dataviewer/NDS to find it.

On further investigation it looks to be some incorrect time stamps at several points in the last few months are causing the problems.  Basically when Alex and I made mistakes in the GPS time stamp settings for the frame builder (daqd) code, the wrong time got written for hours to the raw minute trend data files.

So Alex is going to be running a script to go through the roughly 180 gigabytes of affected trend data to write new files with the correct time stamps.  Once it done, we'll move the files over.  We'll probably lose a few hours worth of recent trend data, depending on how quickly the scripts run, but after which minute trends should work as they are supposed to.

I updated our lockin simulink pieces to use the newer, more streamlined lockin piece that is currently in CDS_PARTS (with new documentation block!).  It means we are no longer passing clock signals through three levels of boxes.

In order to use the piece, you need to right click on it after copying from CDS_PARTS and go to Link Options->Disable Link.  This forces the .mdl to save all the relevant information about the block rather than just a pointer to the library.  I talked with Rolf and Alex today and we discussed setting up another model file, non-library format for putting generically useful user blocks into, rather than using the CDS_PARTS library .mdl.

The BS, ITMX, ITMY, PRM, SRM, ETMX, ETMY now have working lockins, with the input matrix to them having the 2nd input coming from LSC_IN, the 3rd from the oplev pitch, and the 4th from oplev yaw.

This necessitated a few name changes in the medm screens.  I also changed the lockin clock on/off switch to a direct amplitude entry, which turns green when a non-zero value is entered.

Currently, the Mode cleaner optic suspension screens have white lockins on them.  I started modifying a new set of screens just for them, and will modify the generate_master_screens.  Unfortunately, this requires modifying two sets of suspension screens going forward - the main interferometer optics and the MC optics.

[Joe, Alex]

Yesterday during the day, Alex ran a script to fix the time stamps in the trends files we had messed up back during the daqd change overs around Feb 17th and 23rd.  See this elog for more information on the trend problem.

Due to how the script runs, basically taking all the data and making a new copy with the correct time stamps, the data collected while the script was running didn't get converted over.  So when he did the final copy of the corrected data, it created a several hour gap in the data from yesterday during the day time.

The original files still exist on the fb machine in /frames/trend/minute_raw_22mar2011 directory.

Last Friday, we discovered a bug in the RCG where the delay part was not actually delaying.  We reported this to Alex who promptly put a fix in the same day.  This allowed Matt's newly proposed frequency discriminator to work properly.

It also required a checkout of the latest RCG code (revision 2328), and rebuild of the various codes.  We backed up all the kernel and executables first such as mbuf.ko and awgtpman.

We did the following:

1) Log into the fb machine.

2) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/src/drv/mbuf and run make.  Copy the newly built mbuf.ko file to /diskless/root/modules/2.6.34.1/kernel/drivers/mbuf/mbuf.ko on the fb machine.

3) Use "sudo cp" to copy the newly built mbuf.ko file to /diskless/root/modules/2.6.34.1/kernel/drivers/mbuf/

4) Go to /cvs/cds/rtcds/caltech/c1/core/advLigoRTS/src/gds and run make.

5) Copy the newly built awgtpman executable to /opt/rtcds/caltech/c1/target/gds/bin/

6) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/src/mx_stream/ and run make.

7) Copy the newly built mx_stream executable to /opt/rtcds/caltech/c1/target/fb/

[Koji,Joe]

PART 1:

Koji was unable to build his c1lst model first thing this morning.  Turns out there was  a bug with RCG parser that was introduced on Friday when we did the RCG updates.  We talked Alex who did a quick comment fix.  The diff is as follows:

Index: Parser3.pm
==============================

This was some code to remove TAGs from the .mdl file for some reason which I do not understand at this time.  I will ask tommorrow in person so I can understand the full story.

PART 2:

Koji then rebuilt and started the c1lst process.  This is his new test version of the LSC code.  We descovered (again) that when you activate too many DAQ channels (simply uncommenting them, not even recording them with activate=1 in the .ini file) that the frame builder crashes.  In addition, the c1lsc machine, which the code was running on, also hard crashed.

When a channel gets added to the .ini file (or uncommented) it is sent to the framebuilder, irregardless of whether its recorded or not by the frame builder.  There is only about 2 megabytes per second bandwidth per computer.  In this case we were trying to do something like 200 channels * 16384 Hz * 4 bytes = 13 megabytes per second.

The maximium number of 16384 channels is roughly 30, with little to no room for anything else.  In addition, test points use the same allocated memory structure, so that if you use up all the capacity with channels, you won't be able to use testpoints to that computer (or thats what Alex has led me to believe).

The daqd process then core dumped and was causing all sorts of martian network slowdowns.  At the same time, the c1lsc computer crashed hard, and all of the front end processes except for the IOP on c1sus crashed.

We rebooted c1lsc, and restarted the c1sus processes using the startc1SYS scripts.  However, the c1susfe.ko apparently got stuck in a wierd state.  We were completely unable to damp the optics and were in general ringing them up severely.  We tried debugging, including several burt restores and single path checks.

Eventually we decided to reboot the c1sus machine after a bit of debugging.  After doing a burt restore after the reboot, everything started to damp and work happily.  My best guess is the kernel module crashed in a bad way and remained in memory when we simply did the restart scripts.

After I did several things to add new DAQ channels on c1iscex it suddenly became out of network. Maybe crashed.

Then c1iscex didn't respond to a ping and all the epics values associated with c1iscex became not accessible.

I physically shut it down by pushing the reset button. Then it came back and is now running fine.

(how I broke it)

Since activateDAQ.py has screwed up the 'ini' files including C1SCX.ini, I was not able to add a channel to C1SCX.ini by the usual daqconfig GUI.

So I started editing it in a manual way with an editor and changed some sentences to that shown below

Then I rebooted fb to reflect the new DAQ channels.

After that I looked at the C1_FE_STATUS.adl screen and found some indicator lights were red.

So I pushed "Diag reset" button and "DAQ Reload" button on the C1SCX_GDS_TP.adl screen and then c1iscex died.

After the reboot the new DAQ channels looked acquired happily.

This is my second time to crash a front end machine (see this entry)

Problem:

The controls (fast and slow both) think ITMX is ITMY and ITMY is ITMX.

Solution:

After some poking around today, I have convinced myself it is sufficient to simply swap all instances of ITMX for ITMY in the C1_SUS-AUX1_ITMX.db  file, and then rename it to C1_SUS-AUX1_ITMY.db (after having moved the original C1_SUS-AUX1_ITMY.db to a temporary holding file).

A similar process is then applied to the original C1_SUS-AUX1_ITMY.db file.  These files live in /cvs/cds/caltech/target/c1susaux.  This will fix all the slow controls.

To fix the fast controls, we'll modify the c1sus.mdl file located in /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/simLink/ so that the ITMX suspension name is changed to ITMY and vice versa.  We'll also need to clean up some of the labeling

At Kiwamu and Bryan's request, this will either be done tomorrow morning or on Monday.

So the steps in order are:

1) cd /cvs/cds/caltech/target/c1susaux

2) mv C1_SUS-AUX1_ITMX.db C1_SUS-AUX1_ITMX.db.20110408

3) mv C1_SUS-AUX1_ITMY.db C1_SUS-AUX1_ITMY.db.20110408

4) sed 's/ITMX/ITMY/g' C1_SUS-AUX1_ITMX.db.20110408 > C1_SUS-AUX1_ITMY.db

5) sed 's/ITMY/ITMX/g' C1_SUS-AUX1_ITMY.db.20110408 > C1_SUS-AUX1_ITMX.db

6) models

7) matlab

8) Modify c1sus model to swap ITMX and ITMY names while preserving wiring from ADCs/DACs/BO to and from those blocks.

9) code; make c1sus; make install-c1sus

10) Disable all watchdogs

11) Restart the c1susaux computer and the c1sus computer

Problem:

The daqd process was dying every minute or so when it couldn't write frame.  This was slowing down the network by writing a 2.9G core dump over NFS every minute or so. (In /opt/rtcds/caltech/c1/target/fb/).

The problem was /frames/ was 100% full.

Apparently, when we switched the fb over to Gentoo, we forgot to install crontab and a wiper script.

Solution:

We will install crontab and get the wiper script installed.

[Joe, Jamie, Alex]

Fixes:

I asked Alex which cron to use (dcron? frcron?).  He promptly did the following:

emerge dcron

rc-update add dcron default

Copied the wiper.pl script from LLO to /opt/rtcds/caltech/c1/target/fb/

At that point, I modified wiper.pl script to reduce to 95% instead of 99.7%.

I added controls to the cron group on fb:

sudo gpasswd -a controls cron

I then added the wiper.pl to the crontab as the following line using crontab -e.

0 6 * * *       /opt/rtcds/caltech/c1/target/fb/wiper.pl --delete &> /opt/rtcds/caltech/c1/target/fb/wiper.log

Notes:

Note, placing backups on the /frames raid array will break this script, because it compares the amount in the /frames/full/, /frames/trends/minutes, and /frames/trends/seconds to the total capacity. 

Apparently, we had backups from September 27th, 2010 and March 22nd, 2011.  These would have broken the script in any case. 

We are currently removing these backups, as they are redundant data, and we have rsync'd backups of the frames and trends.  We should now have approximately twice the lookback of full frames.

Problem:

I switched the ITMX and ITMY control channels yesterday, but forgot to update the IFO_ALIGN.adl file (/opt/rtcds/caltech/c1/medm/c1ifo/) which had the control labels swapped to make life easier.

Solution:

I swapped ITMX and ITMY control locations on the screen.

Question:

Are there any other screens involving ITMX and ITMY that had controls reversed to make life easier?

[Jamie, Jenne, Koji]

We installed the new c1lsc and started the process.

We still need to configure bunch of the EPICS variables, matrices, and some of the filters.
This should be done in order to transmit the signals to the suspensions.
Jenne is going to work on this task tomorrow (Friday) morning,
and Koji will take over the task afternoon/evening.

[Joe, Alex]

Problem Symptoms:

There were red lights on the status screen indicating RFM errors for the c1scy, c1mcs and c1rfm processes.

The c1iscey, c1sus machines were receiving data sent over the RFM network from the c1ioo computer with a bad time stamp, a few cycles too late.  The c1iscex computer was receiving data from c1ioo fine.

Problem:

The c1iscex RFM card had gotten into a bad state and was somehow slowing things down/corrupting data.  It didn't affect itself, but due to the loop topology was messing everyone else up.  Basically the only one who wasn't throwing an error was the culprit.

Solution:

Hard power cycling the c1iscex computer reset the RFM card and fixed the problem.

Problem:

The original matrix naming conventions for the front ends was broken.  It used _11, _12,...,_1e, _1f, _110, _111 and so forth.  The code was changes to use _1_1, _1_2,...,_1_16,_1_17, and so on.

In addition the matrix of filter banks was modified to use the same naming convetion (instead of starting at zero, it now start with one).

Work Done:

I rebuilt all the models, and restarted them all.

I wrote a simple script to modify the burt restore files to have the correct names for all the stored matrix values.

I also modified all the suspension screens, by modifying the default screens in /opt/rtcds/caltech/c1/medm/master/

The C1SUS, C1SCX, C1SPX, C1SCY, C1SPY, and C1MCS models had their foton filter files modified to put filters into the newly changed named filters

Just a curiosity:

I just wonder how you have distingushed the difference between _111 and _111.

They are equivalent alone themselves. Have you looked at the contexts of the lines?
Or you just did not have the larger matrix than 16x16, did you?

Problem:

Way back, Jay had D-sub to SCSI adapters made to adapt our existing Sander box AA filters to the new SCSI based IO chassis.  However, these did not fit inside the box.

At the time, we simply left the cards outside hanging, which was a hack and needed to be replaced.

Solution:

Steve modified a black AA filter box so that it could fit the D-sub to SCSI adapter board on it, plus strain relief the SCSI cable, rather than let it hang.  The back of the box was cut, and an extending piece of metal attached to the bottom of the box.  The adapter board was screwed into the box, the SCSI plugged in, then the SCSI cable is clamped to the extending metal as well.

This modification will be propagated to the 3 remaining AA filter boards using the D-sub to SCSI adapter.

What was done:

1) Turn off MC1, MC2, MC3, BS, ITMX, ITMY, PRM, SRM watchdogs.

2) Turn c1sus computer off (sudo shutdown now)

3) Go connect monitor and keyboard to c1sus.  Turn c1sus on.

4) Hit "del" key at the right time to go to setup (BIOS).

5) Go to BIOS advanced tab, CPU options, enable Multi-threading.

6) Hit F10 to save and let the computer continue booting.

What went wrong:

Once c1sus was up, I noticed several red lights and dead keep alives for the c1sus models.

Typing dmesg on c1sus revealed many messages like:

[  107.583420] c1x02: cycle 33737 time 20; adcWait 10; write1 0; write2 0; longest write2 0
[  107.583771] c1x02: cycle 33760 time 19; adcWait 11; write1 0; write2 0; longest write2 0

This indicates the Input/Output Processor (IOP) is not completing its duties within the 15 microseconds (1/64 kHz) that it has.  These lines indicate its take 20 or 19 microseconds.  (I saw messages ranging from 16 to 22 microseconds).

So this seems to agree with Rolf's observations that hyperthreading can cause a 5-10 microsecond increase in computation time.

So the next thing to do is modify which core the codes are running on, and try to get them paired up on the same physical core.

First, I disabled front end starts on boot up, and brought c1sus up.  I rebuilt the models for the c1sus computer so they had a new specific_cpu numbers, making the assumption that 0-1 were one real core, 2-3 were another, etc.

Then I ran the startc1SYS scripts one by one to bring up the models.  Upon just loading the c1x02 on "core 2" (the IOP), I saw it fluctuate from about 5 to 12.  After bringing up c1sus on "core 3", I saw the IOP settle down to about 7 consistently.  Prior to hyper-threading it was generally 5. 

Unfortunately, the c1sus model was between 60 and 70 microseconds, and was producing error messages a few times a second

[ 1052.876368] c1sus: cycle 14432 time 65; adcWait 0; write1 0; write2 0; longest write2 0
[ 1052.936698] c1sus: cycle 15421 time 74; adcWait 0; write1 0; write2 0; longest write2 0

Bringing up the rest of the models (c1mcs on 4, c1rfm on 5, and c1pem on 6), saw c1mcs occasionally jumping above the 60 microsecond line, perhaps once a minute.   It was generally hovering around 45 microseconds.  Prior to hyper-threading it was around 25-28 microseconds.

c1rfm was rock solid at 38, which it was prior to hyper-threading.  This is most likely due to the fact it has almost no calculation and only RFM reads slowing it down.

c1pem continued to use negligible time, 3 microseconds out of its 480.

I tried moving c1sus to core 8 from core 3, which seemed to bring it to the 58 to 65 microsecond range, with long cycles every few seconds.

I built 5 dummy models (dua on 7, dub on 9, duc on 10, dud on 11, due on 1) to ensure that each virtual core had a model on it, to see if it helped with stabilizing things.  The models were basically copies of the c1pem model.

Interestingly, c1mcs seemed to get somewhat better and only taking to 30-32 microseconds, although still not as good as its pre-hyper-threading 25-28.  Over the course of several minutes it was no longer having a long cycle.

c1sus got worse again, and was running long cycles 4-5 times a second.

At this point, without surgery on which models are controlling which optics (i.e. splitting the c1sus model up) I am not able to have hyper-threading on and have things working.  I am proceeding to revert the control models and c1sus computer to the hyper-threading state.

New SVN

We are now using the LIGO CDS SVN for storing our control models.

The SVN is at:

https://redoubt.ligo-wa.caltech.edu/websvn/

The models are under cds_user_apps, then trunk, then approriate subsystem (ISC for c1lsc for example), c1 (for caltech 40m), then models.

We have checked out /cds_user_apps to /opt/rtcds/.

So to find the c1lsc.mdl model, you would go to /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsc.mdl

This SVN is shared by many people LIGO, so please follow good SVN practice.  Remember to update models ("svn update") before doing commits.  Also, after making changes please do an update to the SVN so we have a record of the changes.

New Practices

We are creating soft links in the /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/simLink/ to the models that you need to build.  So if you want to add a new model, please add it to the cds_users_apps SVN in the correct place and create a soft link to the simLink directory.

lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1sus.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1sus.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1sup.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1sup.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1spy.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1spy.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1spx.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1spx.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1scy.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1scy.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1scx.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1scx.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1mcs.mdl -> /opt/rtcds/cds_user_apps/trunk/SUS/c1/models/c1mcs.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x05.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x05.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x04.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x04.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x03.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x03.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x02.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x02.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1x01.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1x01.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1rfm.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1rfm.mdl
lrwxrwxrwx 1 controls controls      55 Apr 28 14:41 c1dafi.mdl -> /opt/rtcds/cds_user_apps/trunk/CDS/c1/models/c1dafi.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1pem.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1pem.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1mcp.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1mcp.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1lsp.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsp.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1lsc.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1lsc.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1ioo.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1ioo.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gpv.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gpv.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gfd.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gfd.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1gcv.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1gcv.mdl
lrwxrwxrwx 1 controls controls      54 Apr 28 14:41 c1ass.mdl -> /opt/rtcds/cds_user_apps/trunk/ISC/c1/models/c1ass.mdl

 The same modification was carried out at 1X5 for PRM & SRM.

Note:  D68L8EX-850Hz  are removed  and bypassed in 7 channels.

1) Filled in the C1SUS_BS_OLMATRIX properly so as to make the BS oplev work for Steve.

2) Turned on the ITMX damping.  Apparently it had tripped this morning, possibly due to work in the lab area.

3) The ETMX FE controller (c1scx) had ADC timed out and died sometime around 8:30 am.  The c1x01 (the IOP on the ETMX computer) was also indicating a FB status error (mismatch in DAQ channels).

The reported error in dmesg on c1iscex was:

[1628690.250002] c1spx: ADC TIMEOUT 0 3541 21 3605
[1628690.250002] c1scx: ADC TIMEOUT 0 3541 21 3605

Just to be safe, I rebuilt the c1x01 and c1scx models, ran ./activateDAQ.py, and used the scripts killc1spx, killc1scx, and killc1x01.

I finally restarted the process with startc1x01, startc1scx, and startc1spx.  Everything is currently alive and indicating all green.