40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 36 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  3668   Thu Oct 7 14:57:52 2010 josephb, yutaUpdateCDSc1sus status

Around noon, Yuta and I were trying to figure out why we were getting no signal out to the mode cleaner coils.  It turns out the mode cleaner optic control model was not talking to the IOP model. 

Alex and I were working under the incorrect assumption that you could use the same DAC piece in multiple models, and simply use a subset of the channels.  He finally went and asked Rolf, who said that the same DAC simulink piece in different models doesn't work.  You need to use shared memory locations to move the data to the model with the DAC card.  Rolf says there was a discussion (probably a long while back) where it was asked if we needed to support DAC cards in multiple models and the decision was that it was not needed.

Rolf and Alex have said they'd come over and discuss the issue.

In the meantime, I'm moving forward by adding shared memory locations for all the mode cleaner optics to talk to the DAC in the c1sus model.


Note by KA: Important fact that is worth remembering

  3671   Thu Oct 7 16:21:02 2010 KojiOmnistructureCDSBig 3

Both Rolf and Alex (at least his elbow) together visited the 40m to talk with Joe for the CDS.

40m is the true front line of the CDS development!!!

  3673   Thu Oct 7 17:19:55 2010 josephb, alex, rolfUpdateCDSc1sus status

As noted by Koji, Alex and Rolf stopped by.

We discussed the feasibility of getting multiple models using the same DAC.  We decided that we infact did need it. (I.e. 8 optics through 3 DACs does not divide nicely), and went about changing the controller.c file so as to gracefully handle that case.  Basically it now writes a 0 to the channel rather than repeating the last output if a particular model goes down that is sharing a DAC.

In a separate issue, we found that when skipping DACs  in a model (say using DACs 1 and 2 only) there was a miscommunication to the IOP, resulting in the wrong DACs getting the data.  the temporary solution is to have all DACs in each model, even if they are not used.  This will eventually be fixed in code.

At this point, we *seem* to be able to control and damp optics.  Look for a elog from Yuta confirming or denying this later tonight (or maybe tomorrow).


  3675   Thu Oct 7 23:24:44 2010 yutaUpdateCDSchecking MC1 suspension damping

 The new CDS is currently being set up.
 We want to see if the damping servo of the suspensions are working correctly.
 But before that, we have to see if the sensors and the coils are working correctly.
 Among the 8 optics, MCs take top priority because of the green beam. for the alignment of the in-vac optics.

What I did:
 By seeing the 5 sensor outputs (C1:SUS-MC1_XXSEN_IN1, XX=UL,UR,LR,LL,SD) with the Data Viewer, I checked if all the coils are kicking in the supposed direction and all the sensors are sensing that kick correctly.

 All the matrices elements were set to the ideal values(-1 or 0 or 1) this time.

 They were perfect.
1. POSITION seemed to be POSITION
 When the offset(C1:SUS-MC1_SUSPOS_OFFSET) was added, all the sensor output moved to the same direction.
2. PITCH seemed to be PITCH
 When the offset(C1:SUS-MC1_PIT_COMM) was changed, UL&UR and LL&LR went to the different direction.
3. YAW seemed to be YAW
 When the offset(C1:SUS-MC1_YAW_COMM) was changed, UL&LL and UR&LR went to the different direction.
4. SIDE seemed to be SIDE
 When the offset(C1:SUS-MC1_SDSEN_OFFSET) was added, DC level of the SD sensor output changed.


 c1mcs crashed many times during the investigation, and I had to kill and restart it again and again.
 It seemed to be easily crashed when filters are on, and so I couldn't check whether the damping servo is working correctly or not today.

Next work:

  - fix c1mcs (and maybe others)
  - check the damping servo by comparing the displacements of each 4 degrees of freedom when servo in off and on.

  3678   Fri Oct 8 12:21:11 2010 josephbUpdateCDSchecking MC1 suspension damping

Upon investigation, it appears that the c1mcs model was (and still is) timing out after a random amount of time. Or in other words, it at some point it was taking too long to do all the calculations for a single cycle and falling behind. The evidence for this is from the dmesg command when run on c1sus.

There's a bunch of lines like:

[ 8877.438002] c1mcs: cycle 568 time 62; adcWait 0; write1 0; write2 0; longest write2 0

[ 8877.438002] c1mcs: cycle 569 time 62; adcWait 0; write1 0; write2 0; longest write2 0

With a final line like: [ 8877.439003] c1mcs: ADC TIMEOUT 1 2405 37 2277

This last line indicates in fell so far behind it gave up.

However, this doesn't actually seem to be related to the amount of computation going on with the front end. I restarted the c1mcs model this morning by logging into the c1sus machine, and changing to the /opt/rtcds/caltech/c1/target/c1mcs/bin directory and running:


sudo rmmod c1mcsfe

sudo insmod c1mcsfe.ko

The first line just lists the running modules. The second removes the c1mcs module, and the third starts it up again.

I proceeded to turn all the filters and and set all the matrix values while keeping an eye on the C1MCS-GDS_TP.adl screen and its timing indicator. It was running fine with all these turned on. Then I ran a dtt session from rosalba (going to /opt/apps/, typing bash, then source gds-env.bash, and finally diaggui) as well as a dataviewer and looked at 6 test point channels. It received data fine.

However, about 2 minutes after I had stopped doing this (although the dataviewer realtime session was still going) the USR timing jumped from about 20 microseconds to 35 microseconds, and the CPU Max timing jumped to the 50-60 microsecond range. At this point dmesg started reporting things like:

[54143.465613] c1mcs: cycle 1076 time 62; adcWait 0; write1 0; write2 0; longest write2 0

[54143.526004] c1mcs: cycle 2064 time 62; adcWait 0; write1 0; write2 0; longest write2 0

About a minute later the code gave up and reported a ADC timeout via dmesg. None of the other front ends seem to be affected.

I had to clear the test points manually after stopping dataviewer and dtt by going to rosalaba,using the sourced gds-env.bash, and running diag -l. I then typed "tp clear 36 *" to clear all the test points on the model with FEC number 36 (corresponding to c1mcs).

I removed and restarted c1mcs again, trying to turn on a few things at a time and letting it run for a few minutes to see if I could narrow down if its one particular filter perhaps reaching an underflow and starting to bog down the computations. However, after about 45 minutes of this, the model is still running and I've turned all the filters on and have been running about 8 test points with no problem, so the problem is clearly intermittent.


 c1mcs crashed many times during the investigation, and I had to kill and restart it again and again.
 It seemed to be easily crashed when filters are on, and so I couldn't check whether the damping servo is working correctly or not today.

Next work:

  - fix c1mcs (and maybe others)
  - check the damping servo by comparing the displacements of each 4 degrees of freedom when servo in off and on.


  3680   Fri Oct 8 15:57:32 2010 josephbUpdateCDSc1ioo status

I've been trying to figure out why the c1ioo machine crashes when I try to run the c1ioo front end.

I tried removing some RFM components from the c1ioo model, and then the c1gpt model (Kiwamu's green locking model) as an easier test case.  Both cause the machine to lock up once they start running.  Lastly, I tried running the c1x02 and c1sus models on the c1ioo machine instead of the c1sus machine, after first turning off all models on c1sus.  This led to the same lockup. 

Since those models run fine on the c1sus machine, I could only conclude that a recent change in the fe code generator or the Gentoo kernel and the Sun X4600 computer don't work together at the moment.

After talking to Alex, he got the idea to check if the monitor() and mwait() were supported on the c1ioo machine.  These function calls were added relatively recently, and are   used to poll a memory location to see if something has been written there, and then do something when it is.  Apparently the Sun X4600 computers do not support this call.  Alex has modified the code to not use these functions calls, at least for now.  He'd like to add a check to the code so it does use those calls on machines which have them supported.

After this change, the c1ioo and c1gpt front end codes do in fact run.

  3681   Fri Oct 8 17:35:24 2010 josephbUpdateCDSstatus of c1ioo, c1sus and rfm

RFM is still not working.  I can see data on a filter just before it reaches the RFM sending code, but I see only zeros on the receiving side.

c1sus machine and c1x02, c1sus, c1mcs, c1rms are running.  At the moment, the c1mcs model is running at about 42 microseconds for USR time and 56 microseconds for CPU MAX, which is close to the 61 microsecond limit.  This is with MC filters on.  So far it has not been late, but its not clear to me if its going to stay that way.  So far I haven't been able to isolate why it sometimes slows down after a few minutes.  Also, it was running faster earlier in the day (around 30-ish microseconds) and I believe it has slowed down slightly in the last hour or two.

c1ioo machine and c1x03, c1ioo are running. However its not doing very much good as I can't get any data transferred from it to any of the optic suspensions. I need to spend some more time debugging this and then grab Alex I think.

  3687   Mon Oct 11 10:49:03 2010 josephbUpdateCDSc1sus stability

Taking a look at the c1sus machine, it looks as if all of the front end codes its running (c1sus - running BS, ITMX, ITMY, c1mcs - running MC1, MC2, MC3, and c1rms - running PRM and SRM) worked over the weekend.  As I see no

Running dmesg on c1sus reports on a single long cycle on c1x02, where it took 17 microseconds (~15 microseconds i maximum because the c1x02 IOP process is running at 64kHz).

Both the c1sus and c1mcs models are running at around 39-42 microseconds USR time and 44-50 microseconds CPU time.  It would run into problems at 60-62 microseconds.

Looking at the filters that are turned on, it looks as it these models were running with only a single optic's worth of filters turned on via the medm screens.  I.e. the MC2 and ITMY filters were properly set, but not the others.

The c1rms model is running at around 10 microseconds USR time and 14-18 microseconds CPU time.  However it apparently had no filters on.

It looks as if no test points were used this weekend.  We'll turn on the rest of the filters and see if we start seeing crashes of the front end again.


The filters for all the suspensions have been turned on, and all matrix elements entered.  The USR and CPU times have not appreciably changed.  No long cycles have been reported through dmesg on c1sus at this time.  I'm going to let it run and see if it runs into problems.

  3690   Mon Oct 11 17:31:44 2010 yutaUpdateCDSActivation of DAQ channels for 8 optics

(Joe, Yuta)

 We need DAQ channels activated to measure Q-values of the ringdowns for each DOF, each optics with the Dataviewer.

What we did:
1. Activated the following DAQ using daqconfig (in /cvs/cds/rtcds/caltech/c1/scripts).

2. Set datarate to 2048 for each DAQ.

3. Restarted fb(frame builder).

 We succeeded in making DAQ channels appear in the Dataviewer signal list, but we can't start the measurement because c1mcs is still flaky.

 We found that c1mcs crashes everytime when turning off all the damping servo (using "Damp" buttons on the medm screen).
 It doesn't crash when all the filters are off.

  3691   Mon Oct 11 20:52:00 2010 ranaUpdateCDSActivation of DAQ channels for 8 optics

Bah!  We tried to get some data but totally failed. It seems like there is some rudimentary functionality in the FE process of the MC, but no useful DAQ at all.

Neither Dataview or DTT had anything. We looked and the NDS process and the DAQD process were not running on FB.

I restarted them both (./daqd -c daqdrc) & (./nds pipe > nds.log) but couldn't get anything.

There's a bunch of errors in the daqd.log like this:

    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-MC1_SUSPOS_INMON", Connecting to: c1susdaq:57416, Ignored: c1sus.martian:57416"
    Source File: ../cac.cpp line 1208
    Current Time: Mon Oct 11 2010 18:25:15.475629328
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-MC1_SUSPIT_INMON", Connecting to: c1susdaq:57416, Ignored: c1sus.martian:57416"
    Source File: ../cac.cpp line 1208
    Current Time: Mon Oct 11 2010 18:25:15.475900124

  3692   Mon Oct 11 22:04:28 2010 yutaUpdateCDSdamping for MCs are basically working

 Even if we can't use the Dataviewer to get the time series of each 4 DOF displacements, we can still use StripTool to monitor the ringdowns and see if the damping servo is working.

What I did:
1. Excited vibration by kicking the mirror randomly (by putting some offsets randomly and turing the filters on and off randomly).

2. Turned all the servo off by clicking "ShutDown" button.

3. Turned all the servo on by clicking "Normal" button.

3. Monitored each 4 DOF displacements with StripTool and see if there are any considerably low-Q ringdown after turning on the servo.
 The values I monitored are as follows;

All the settings I used for this observation are automatically saved here;

 Attached is the screenshots of StripTool Graph window for each 3 MCs.
 As you can see, the dampings for each DOF, each MCs are basically working.

 Do NOT turn off all the damping servo by clicking "Damp" buttons or setting the SUSXXX_GAIN to 0. It crashes c1mcs.

Next work:
 - check and relate the signal sign with the actual moving direction of the optics
 - fix data aquisition system
 - measure Q-values when the servo is on and off using DAQ and Dataviewer

  3696   Tue Oct 12 13:05:00 2010 josephb, alexUpdateCDSfb still flaky, models time out fixed

Interesting information from Alex.  We're limited to 2 Megabytes per second per front end model.  Assuming all your channels are running at a 2kHz rate, we can have at most 256 channels being set to the frame builder from the front end (assuming 4 byte data).  We're fine for the moment, but perhaps useful to keep in mind.

I talked to Alex this morning and he said the frame builder is being flaky (it crashed on us twice this morning, but the third time seemed to stay up when requesting data).  I've added a new wiki page called "New Computer Restart Prodecures" under Computers and Scripts, found here. It includes all the codes that need to be running, and also a start order if things seem to be in a particularly bad state.  Unfortunately, there were no fixes done to the frame builder but it is on Alex's list of things to do.

In regards to the timing out of the front ends, Alex came over to the 40m this morning and we sat down debugging.  We tried several things, such as removing all filters from C1MCS.txt file in the chans directory, and watching the timing as we pressed various medm control buttons. We traced it to a filters used by the DAC in the model talking to the IOP front end, which actually sends the data to the physical DAC card.  The filter is used when converting between sample rates, in this case between the 16 kHz of the front end model and the 64 kHz of the IOP.  Sending it raw zeros after having had real data seemed to cause this filter to eat up an usually large amount of CPU time. 

We modified the /opt/rtcds/caltech/c1/core/advLigoRTS/src/include/drv/fm10Gen.c file.

We reverted a change that was done between version 908 and 929, where underflows (really small numbers) were dealt with by adding and then subtracting a very small number.  We left the adding and subtracting, but also restored the hard limits on the history.

So instead of relying on just:

input += 1e-16;
junk = input;
input -= 1e-16;

we also now use

if((new_hist < 1e-20) && (new_hist > -1e-20)) new_hist = new_hist<0 ? -1e-20: 1e-20;

Thus any filter value who's absolute value is less than 1e-20 will be clamped to -1e-20 or 1e-20.  On the bright side, we no longer crash the front ends when we turn something off.


  3706   Wed Oct 13 17:04:34 2010 josephb, yutaUpdateCDSburt restore now working with filter settings

Previously, burt restoring was not setting filters on and off, which was required us to constantly go through all the filter banks and turn them on.  We figured it out that the autoBurt.req file doesn't seem to be setup to restore those values, so that snapshots made with the default .req file don't restore either.

So I went to each of the suspension directories (/opt/rtcds/caltech/c1/target/c1SYS/c1SYSepics/   where SYS is sus, mcs, or rms) and modified teh autoBurt.req files found there with the following incantation:

sed -i 's/RO \(.*SW[12]R.*\)/\1/' autoBurt.req

This removes the RO at the beginning of the lines which have SW1R or SW2R in them.  These are the channels which correspond to filter bank switches.  As far as I can tell, the RO means to leave it alone.  Unfortunately, I didn't see an obvious autoBurt file specification in the DCC or on google in the 2 minutes I took to look for it.

We need to talk to Alex to figure out how that autoBurt.req file is generated and get it so it doesn't default to not restoring filter bank settings.

  3707   Wed Oct 13 17:12:33 2010 josephb, yutaUpdateCDSFilter name length problem found and fixed

The missing filter files for ULPOS, URPOS, and so forth for the mode cleaner optics was due to the length of the names of the filters. 

This was not a problem for the c1sus model because it was using its own name as the first 3 letters of its designation.  A filter for the sus model would be called something like BS_TO_COIL_MTRX_0_0, while for the mcs it would be called SUS_MC1_TO_COIL_MTRX_0_0, an extra 4 characters.

However, the c1mcs model used the "top_name" feature which uses a subsystem box within the simlink model to rename all the channels.  Apparently in the filter file, this means it has to add the top name to the front of everything, adding an additional 3 characters.  This pushed things over the length limit.

A hard cap of 18 characters has been added to the FiltMuxMatrix.pm file (located in /cvs/cds/caltech/c1/, so that it will prevent this type of problem in the future by stopping at compile time and presenting a helpful error message.

I also fixed a bug with too many spaces in the feCodeGen.pl file when dealing with top_names and the filtMuxMatrix.pm preventing some .adl files from being generated.

Also of interest, MC3 appears to never have had F2A filters.  For the moment we're running without them, but since they're just a fine tuning it shouldn't affect locking tonight.


Improbability factor of mode cleaner suspensions working tonight: ~20


  3708   Wed Oct 13 18:01:43 2010 yutaHowToCDSeditting all the similar medm screens

(Joe, Yuta)

Say, you want to edit all the similar medm screens named C1SUS_NAME_XXX.adl.

1. Go to /opt/rtcds/caltech/c1/medm/master and edit C1SUS_DEFAULTNAME_XXX.adl as you like.

2. Run generate_master_screens.py.

That's it!!

  3712   Thu Oct 14 00:54:12 2010 ranaHowToCDSBad PSL Quad, so I edited the SUS MEDM screens

We found today that there was some (unforgivably un-elogged) PSL-POS & PSL-ANG work today.

The wedge splitter at the end of the PSL table is so terribly dogged that it actually moves around irreversibly if you touch the post a little. Pictures have been recorded for the hall of shame. This should be replaced with a non-pedestal / fork mount.

I was so sad about the fork clamp, that I edited the SUS Master screen instead according to Joe-Yuta instructions; see attached.Untitled.png

There's clearly several broken/white links on there, but I guess that Yuta is going to find out how to fix these from Joe in the morning.

  3720   Thu Oct 14 14:09:30 2010 josephbUpdateCDSNDS 2 server status

[Joe, John]

The nds 2 server is about 50% of the way there.  You can connect to it and get channel lists, but its having issues actually serving the data.  The errors we're getting basically say it can't find the source data for the channel

John had to go get lunch at 2pm, but he said he'd log in remotely later and try to configure it properly.

  3728   Fri Oct 15 15:32:35 2010 josephbUpdateCDSslow DAQ channels added

I added all the _OUT16, _INMON, _EXCMON channels associated with the BS, ITMX, and ITMY channels in SUS_SLOW.ini in the /opt/rtcds/caltech/c1/chans/daq directory.  Similarly, I added the channels associated with MC1, MC2, and MC3 to MCS_SLOW.ini and those associated with PRM and the SRM to RMS_SLOW.ini. 

Lines pointing to these files were then added to the /opt/rtcds/caltech/c1/target/fb/master file and the frame builder restarted.  It took about 4 tries before the frame builder stayed up using ./daqd -c ./daqdrc.

I generated the .ini files with a python script.  Its at /opt/rtcds/caltech/c1/scripts/daqscripts/create_inmon_out16_daq_ini.py. It checks the C0EDCU.ini to find what slow channels already exist, then goes through a medm directory given at the command line looking for all _OUT16, _INMON, and _EXCMON channels, adding them if they aren't already accounted for, and then writing out the new file to the location of your choice.

  3732   Mon Oct 18 08:44:36 2010 ranaUpdateCDSslow DAQ channels added

Why EXCMON? I think that we should modify the filter coefficients so that the decimation filter that makes OUT16 is a 2nd order Butterworth at 1 Hz instead of whatever bogus thing it is now. Then we can delete the EXCMON from the DAQ and add either OUTPUT or OUTMON. That way we will have a low frequency signal (OUT16) and a sort of aliased RMS (OUTMON).

  3735   Mon Oct 18 15:33:00 2010 josephb, alexUpdateCDSPossibly broken timing cards

It looks as though we may have two IO chassis with bad timing cards.

Symptoms are as follows:

We can get our front end models writing data and timestamps out on the RFM network.

However, they get rejected on the receiving end because the timestamps don't match up with the receiving front end's timestamp.  Once started, the system is consistently off by the same amount. Stopping the front end module on c1ioo and restarting it, generated a new consistent offset.  Say off by 29,000 cycles in the first case and on restart we might be 11,000 cycles off.  Essentially, on start up, the IOP isn't using the 1PPS signal to determine when to start counting.

We tried swapping the spare IO chassis (intended for the LSC) in ....

# Joe will finish this in 3 days.

# Basically, in conclusion, in a word, we found that c1ioo IO chassis is wrong.

  3742   Tue Oct 19 21:10:27 2010 yutaUpdateCDSbad filter coefficients for every suspensions!

(Koji, Yuta)

 The damping servos of the MC suspensions has not been working since yesterday. They had worked on Friday.
 We found that some of the filter coefficients somehow changed from the correct value during dealing with the sampling frequency shift (from 2048Hz to 16384Hz). (see elog #3659)

What we did:
We thought that the problem occurred from the CDS update on Monday. So, we restored them using svn.
 1. Went to /opt/rtcds/caltech/c1/core/advLigoRTS and ran svn update using the revision number 2125.
   svn update -r 2125
 2. Rebuilt all the front ends.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/core/advLigoRTS
   make clean-SYSNAME
   make SYSNAME
   make install-SYSNAME 
   (where SYSNAME=c1x02,c1sus,c1mcs,c1rms)
 3. Restarted the front end models.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/scripts
   sudo startc1SYS 
   (where SYS=sus,mcs,rms)
 4. Restarted mx_streams.
   sudo /etc/restart_streams
See this wiki page for the restart procedures.

At this point we judged that the realtime system and "dataviewer" worked fine (by poking some suspensions / by measuring PSDs/TFs of the signals).

However, just restoring the RT system didn't fix the damping servos.

After that, we measured the transfer functions of the servo filters using DTT(diaggui on rosalba).
 5. The filter at MC1_SUSPOS, "3:0.0" should have a cut-off frequency at 3Hz, but it was around 0.3Hz.
 6. We used foton in order to look at the filter bank files (in /cvs/cds/rtcds/caltech/c1/chans). Then we found that the filter design was somehow changed to
  instead of the correct one

Why the filter designs changed?:
 We suspect that the filter designs were changed because of the replacement of sampling frequency in filter bank files(see elog #3659). We only replaced the lines like
 and didn't changed the coefficients.
 This may caused a problem when opening the files with foton, and foton changed 3Hz to 0.375Hz (2048/16384) and other coefficients.

 The damping servo for SIDE has been working for every MCs. POS, PIT, YAW are the ones not working.

Next work:
 - Fix filters for every suspensions.
   There are backup files in /cvs/cds/rtcds/caltech/c1/chans/filter_archive directory, so this should help.
   But I don't think just fixing filters would fix the damping servo because the filter design of MC suspensions were changed this morning and the damping had worked until Friday.

  3744   Wed Oct 20 01:22:18 2010 yutaUpdateCDSfixed filters for MCs, but no damping

(Rana, Yuta)


 The damping servo for MCs has not been working since Monday.
 We already found that some filter coefficients were wrong.(see elog #3742)
 So, we fixed it (just for MCs now).
 But still doesn't work.

What we did:
Fixed the filters for MC suspensions;
 1. In /cvs/cds/rtcds/caltech/c1/chans/ directory, we replaced C1MCS.txt with ./filter_archive/c1mcs/C1MCS_101019_101927.txt.
  This is the archive of the filter bank for MC suspensions, when filter designs were correct(when we hadn't opened it with foton yet).

 2. Deleted all the filter coefficients.
   By using emacs magical C-<space> C-x r k.

 3. Loaded with foton to get correct coefficients.

 4. Reloaded coefficients for C1MCS.

We now have the correct filters, but the damping still doesn't work.

 5. To confirm that the filters are now correct and the model is working correctly, we measured the transfer function between ULSEN input and ULCOIL output and compared with the calculated TF from the filter bank file (Attachment #1) .

 6. To calculate the designed function, we used the following matlab scripts;
    /cvs/cds/caltech/users/rana/mat/utilities/FotonFilter.m        # takes the foton file and returns the TF
    /cvs/cds/caltech/users/rana/mat/utilities/readFilterFile.m    # reads the foton file
    /cvs/cds/caltech/users/yuta/scripts/sentocoil.m                   # calculates the total TF from SEN to COIL

 As you can see from the attachment, filters seem fine now. (The red line is the measured and the blue line is the calculated function in the plot)
 But the damping (for POS, PIT, YAW) still doesn't work. Why????

Next work:

 - see if coils are pushing correctly
 - push cable connectors further!
 - check input and output filters (whitening, dewhitening)
 - when fixed, use my fully updated QAdjuster.py for adjusting Q for multiple channels automatically!!

  3748   Wed Oct 20 21:43:25 2010 yutaUpdateCDSchecking whitening filters for MCs and messed up

(Joe, Yuta)

 We found that the damping servo for MC suspensions somehow worked when we turned off the 13Hz Chebyshev filters.
 But that does not meet our satisfaction, so we started checking every components.
 First of all is the whitening filters.
 If we turn on a digital whitening filter(WF), corresponding analog WF should turn off, and vice versa.

 See DCC #D000210 for the analog circuit of WF. WF has 3Hz zero, 30Hz pole, 100Hz pole. MAX333A bypasses analog WF when supplied +15V(HIGH).

What we did:
 1. Compared the transfer function between MC2_SUSPOS_EXC and MC2_ULSEN_IN1 when digital WF on and off. When digital is on/off, analog should be off/on, so there should be difference but couldn't see.

 2. We went through the simulink model and found 2 mistakes in the logic. One is the conflict with other optics. Even if we turn on/off digital WF of MC2, it didn't switched analog WF of MC2. Two is the additional bit invert (but it turned out to be our misunderstanding).

 3. We (thought we) fixed it, rebuild it, and measured the TF again.(Attachment #1)

[Attached #1]
 The red/blue line is when digital WF is on/off. Blue should be bigger(+10dB @ 10Hz according to (analog) WTF) than red, but it was the opposite.

 4. To confirm that they are doing the opposite, I checked MAX333A input(pin#1,10,11,20) in "SUS PD Whitening Board" at 1X5-1-8B (which is for MC3) and found that switching is opposite. When I turned off/on the digital WF, MAX333A input was +15V/0V. It should be 0V/+15V.

 5. Also, I found that LLSEN digital WF switch switches LRSEN analog WF and vice versa.

[Attached #2]
 Transfer functions between MC2_SUSPOS_EXC and MC2_LRSEN_IN1 with;
   Red: LR digital off, LL digital on
   Blue: LR digital off, LL digital off
   Green: LR digital on, LL digital off
 As you can see, LL switch is the one which switches LR analog WF now.

 Currently, digital WF on/off corresponds to analog WF on/off.
 Also, LL/LR digital WF switch is LR/LL analog WF switch now.

Next work:
 - fix the simulink models (or wiring)
 - check dewhitening filters

 - whitening
   MC1 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC2 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC3 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
      D000210 has switches for bypassing analog WT(3,100:3). HIGH to bypass.
 - dewhitening
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,3 UL/UR/LR/LL coils
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,2,3 SIDE coils
  (SimDW)(InvDW) ... -> LSC Anti-imaging Board(D000186) -> Universal Dewhitening Board(D000183) -> MC2 UL/UR/LR/LL coils
     D000316 has switches for bypassing 28Hz elliptic LPF. HIGH to bypass.
     D000186 is 7570Hz elliptic LPFs.
     D000183 has switches for bypassing dewhitening filter. HIGH to bypass.
 See this wiki page for more comprehensive setup.

  3757   Thu Oct 21 21:35:16 2010 yutaUpdateCDSfixed whitening filter switches

(Joe, Yuta)

 We found some mistakes in whitening filter switches yesterday(see elog #3748).
 One was the mis-connection between LL/LR and the second was the bit invert.
 So, we fixed it and checked that they are now switching correct by probing MAX333A.

Why we made mistake:
1. LL/LR mis-connection
  Simply, Simulink model was wrong.
  It occurred because UL/UR/LR/LL order is currently arbitrary and confusing.
  For now, we just swap the connection(Attached #1).
  In the future, we should fix all orders corresponding to the actual wiring order(UL > LL > UR > LR).

2. Bit invert
  It occurred from counter-intuitive output of Contec 32 BO(See Attachment #2(taken from this wiki page)).
  If "1," current goes from A to B, so MAX333A input will be 0V, which means analog WF is on.
  If "0," no current, so MAX333A input will be +15V, which means analog WF is off(bypassed).

Next work:
 - There are no digital WF in SDSEN inputs. So, we have to put them.
 - Dewhitening filters are totally messed up now. Switches are wrong, some optics have digital DW but some doesn't, some have 28Hz elliptic LPF.......
     We have to grand unify them.

  3758   Fri Oct 22 00:59:01 2010 yutaUpdateCDSnew MEDM screens, new filter banks(whitening and dewhitening related)

(Joe, Yuta)

 No more discrimination for SIDE!
 We had all SIDE filters in SDSEN, so we splitted into SDSEN, SUSSIDE, and SDCOIL just like other DOFs. (Not SIDECOIL. SDCOIL from now on!)
 Also, there was a confusion in output filters, so we fixed the filter bank(dewhitening and 28HzELP).
 More than that, I found that "13HzCheby" in SUSPOS, SUSPIT, SUSYAW was actually doing ~1.6Hz Chebyshev, so I fixed it. (No wonder we had no damping when turing "13HzCheby" on!)
 Our new MEDM screen is Attachment #1.

Input filter design:
 Every optics, every OSEM has same analog whitening filter.

digital analog
30,100:3 30,100:3

Output filter design:
 For every SIDE coils and MC1, MC3 coils, they have analog 28Hz elliptic LPFs.
digital analog

 For MC2(and other main optics) UL/UR/LR/LL coils, they have analog dewhitening filters.
digital analog

 InvDW and SimDW were in FM10 and FM9, but I swapped them to make it more consistent.
 Now, FM10 switches analog/digital output filiters for every optics.

  Let's put 28HzELP in FM9 so that FM9 switches analog/digital for every optics.


 - whitening
   MC1 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC2 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC3 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
      D000210 has switches for bypassing analog WT(3,100:3). HIGH to bypass.
 - dewhitening
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,3 UL/UR/LR/LL coils
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,2,3 SIDE coils
  (SimDW)(InvDW) ... -> LSC Anti-imaging Board(D000186) -> Universal Dewhitening Board(D000183) -> MC2 UL/UR/LR/LL coils
     D000316 has switches for bypassing 28Hz elliptic LPF. HIGH to bypass.
     D000186 is 7570Hz elliptic LPFs.
     D000183 has switches for bypassing dewhitening filter. HIGH to bypass.
 See this wiki page for more comprehensive setup.

Next work:
 - Fix Simulink connection and logic so that analog/digital switching go correctly.
 - Check if the switching are correct for all channels by measuring transfer functions.
 - Currently, we are focusing on MCs, but we have to do the same fixing for other optics, too.
 - More than setting the right filter TF, there are switching settings like Zero History, Ramp, Zero Crossing...... We should investigate that.
 - Actually, TF of analog DW in D000183 doesn't look like the same with digital SimDW. Why??

  3765   Fri Oct 22 19:53:27 2010 yutaSummaryCDSconversion failure in digital filters

(Rana, Joe, Yuta)

We now understand that we never succeeded in converting old fiter files to new filter files.

For example, we just changed the sampling rate with coefficients remained and foton confused, or we forgot to rename some of the module names(ULSEN -> SUS_MC1_ULSEN) ......

This is why we sometimes damped and sometimes didn't, depending on filter switches. New filter files has been always wrong.

So, we started to convert them again.
We have to figure out how to convert the files that Foton accepts correctly.

  3767   Fri Oct 22 23:29:48 2010 yutaUpdateCDSfixed output filter switching

(Joe, Yuta)

 This evening, we fixed output filter analog/digital switchings in Simulink logic, but they didn't actually switched filters.
 That was because what we thought BO0 was BO2 and BO2 was BO0.
 We re-wired, re-labeled the cables.
 We checked it by using a LED(see "test configuration" in this wiki page).
 Also, we checked the Simulink connection by probing MAX333A in SOS Dewhitening Board located at 1X4-8-11B, which switches SIDE coils for all optics.

Next work:

 - Currently, FM10 in UL/UR/LR/LL/SDCOIL filters switch analog/digital, but let's make FM9 do the switching.(See schematic in elog #3758)
 - Check all the logic by measuring transfer functions(maybe single frequency measurement is enough. we can use tdsdmd etc to automate).

  3769   Sat Oct 23 03:36:05 2010 yutaUpdateCDSfixed filters for C1SUS, C1RMS, C1MCS

(Joe, Yuta)


 This Monday, MC suspension damping got something wrong.
 We started to check filters and found that digital filters were wrong because of mis-conversion from old filter files to new files.
 We converted the file again, and with mutual understanding between Foton and us, we finally got correct filters(I hope!).

What we did:
 1. Merged filter files in old /cvs/cds/caltech/chans/ directory into C1SUS.txt(BS,ITMX,ITMY), C1RMS.txt(PRM,SRM), C1MCS.txt(MC123).

 2. Rebuilt the RT models in order to get a correct filter file header(they have list of filter modules).

 3. Concatenate the header with filter design part which we got from step1.

 4. Replaced 'N 2048' with 'N 16384'
   It replaces sampling rate of "XXSEN"s.

Basically, step1-4 was the same with what we did last time. We didn't changed the fitler coefficients, so Foton somehow changed the original filter design.
So, this time, we

 5. Deleted coefficients like we did on Tuesday (see elog #3774).

But Foton couldn't read the file correctly this time. Foton seemed to be unbeatable.
Even if we replaced the sampling rate, Foton kept saying 2048! (This is maybe because Foton's default value is 2048Hz. Everytime Foton notice some editting in the file, he destroys everything. He hates editting)
The problems were always associated with the sampling rate, so

 6. Got back to step4 and undo-ed the replacement.

 7. Foton could read it this time, so I changed the sampling rate one by one using Foton GUI.

 8. Checked filters using Foton's Bode Plot.(Not for all, but some that had problem before)

 9. Splitted SDSEN filters to SDSEN, SUSSIDE, and SDCOIL.

 10. Put some missing whitening filters, and 28HzELPs.
   BS, PRM, SRM didn't have any 28HzELP for SDCOIL.
   ITMX and ITMY SDCOIL had SimDW/InvDW which doesn't make sense(SIDEs don't have analog DW). So, I deleted and replaced with 28HzELP.

F2A issue:
 We failed in sending F2A filters to new filter files.
 These are a little bit complicated because TO_COIL_X_X filters were named ULPOS,URPOS etc before.
 Also, MC3 didn't have any F2As, so maybe we should but the same F2As as MC1/2.
 Note that every F2As are different, and TO_COIL matrix have UL,UR,LL,LR order(not same as INMATRIX).
 Also, SRM had f2pv instead of F2A!

Next work:
 - Check whole filters by actually measuring transfer function between SENs and COILs.
 - Damp MC suspentions, and lock MC.
 - Measure openloop TF and compare with the designed.

How do you read a Foton filter file:
 When you open up a Foton filter file, you see filters like this.

### modulename                                                               ###
# SAMPLING modulename samplingrate
# DESIGN   modulename n filterdesign
# DESIGN   modulename n filterdesign
###                                                                          ###
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
                                                                            a1     a2    b1     b2
n: filter number
 0 for FM1, 1 for FM2, ... , 9 for FM10

x: Input Switching setting
 1 Always On
 2 Zero History

y: Output Switching setting
 1 Immediately
 2 Ramp
 3 Input Crossing
 4 Zero Crossing

z: number of filters cascaded.

v: if y=2, (Ramp Time(sec))*(samplingrate)
   if y=3 or 4, Tolerance

w: (Timeout(sec))*(samplingrate)

 Note that v and w are changed when sampling rate is changed.

 Transfer function will be;

 where fs is the sampling frequency.

 Kiwamu Izumi: "Notes about Digital Filters," http://tamago.mtk.nao.ac.jp/izumi/green/DigitalFilter.pdf

  3770   Sat Oct 23 14:42:01 2010 yutaSummaryCDSdamped MC suspensions

After replacing filters, MC suspensions damped  last night.

Further measurement next time.

  3778   Mon Oct 25 20:20:59 2010 yuta, JoeUpdateCDSDAQ channel number etc
We split the old SDSEN filters to SDSEN, SDSIDE, SDCOIL last week.
Along with this change, the TP channel number changed unfortunately.
So, we fixed them.
Also, we made FM9 do the output filter analog/digital switching.

What we did:
1. Changed the Simulink logic so that FM9 do the output filter switching, and checked the logic by probing
MAX333A for SDCOILs.

2. After making a new Simulink model and rebuilding, run the following incantation to burt restore filter
settings in /opt/rtcds/caltech/c1/target/c1SYS/c1SYSepics/ (See elog #3706)
sed -i 's/RO \(.*SW[12]R.*\)/\1/' autoBurt.req

3. DAQ channel numbers are listed in C1SYS.ini files in /cvs/cds/rtcds/caltech/c1/chans/daq/.
Channels with # signs are not activated. So, we changed, for example,




for all channels we want to look at.

4. Restarted fb.

- measure TFs and see if input and output filter switchings are working correctly
- make a switch that does all filter switching for all 5 OSEMS or 5 COILS
- put optical lever stuff
- fix offset sliders and offset switch
- put F2A filters in TO_COIL matrix (see elog #3769)
- make a nice graphical screen for MCs (like /cvs/cds/caltech/medm/c1/ioo/C1IOO_ModeCleaner.adl)
- write a script that activates DAQ we need
- make a plan

* SYS=SUS, RMS, MCS, etc
  3785   Tue Oct 26 12:45:20 2010 yutaHowToCDSmaking a new master file for medm screens

(Joe, Yuta)

 In elog #3708, we showed how to edit all the similar medm screens easiliy.
 But if there are no master files you want to edit in /opt/rtcds/caltech/c1/medm/master/ directory, you can make one.
 Say, you want to edit all the similar medm screens named C1SUS_NAME_XXX.adl.

1. Copy one of C1SUS_NAME_XXX.adl to /opt/rtcds/caltech/c1/medm/master/ directory as C1SUS_DEFAULTNAME_XXX.adl.

2. Go to that directory and

3. Add "_XXX.adl" to file_array list in generate_master_screens.py


5. Run ./generate_master_screens.py

That's all!!

  3787   Tue Oct 26 16:02:55 2010 yutaHowToCDSthings to do after making a new Simulink model


Things to do after making a new Simulink model.

1. ssh c1sus, go to /opt/rtcds/caltech/c1/core/advLigoRTS/ and run
 bash ./makestuff.sh c1SYS

 makestuff.sh does;

  make uninstall-daq-$*
  make clean-$*
  make $*
  make install-$*
  make install-daq-$*
  make install-screens-$*
  sed -i 's/RO \(.*SW[12]R.*\)/\1/' /opt/rtcds/caltech/c1/target/$*/$*epics/autoBurt.req

 If you don't need to re-install DAQs or screens, just run line 2,3,4, and 7 and go to step #6.

2. ssh c1sus, go to /opt/rtcds/caltech/c1/scripts/ and run
 sudo ./startc1SYS

 For now, you have to put "1" in "BURT Restore" in GDS screens with in 5-10 secs.

3. Now the DAQ channel numbers are changed. So, go to /cvs/cds/rtcds/caltech/c1/chans/daq/ and run

 activateDAQ will activate the following DAQ channels for every optics with data rate 2048Hz;

4. Restart fb. See this wiki page.
 Basically you what have to do is kill and restart daqd in fb and restart mx_streams in c1sus.

5. DONE! Burt restore if you want.

6. If you don't need to re-install DAQs or screens;
  a. Go to /opt/rtcds/caltech/c1/target/c1SYS/c1SYSepics and run
    sudo rmmod c1SYSfe

  b. Go to /opt/rtcds/caltech/c1/target/c1SYS/bin/ and run
    sudo insmod c1SYSfe.ko

  3788   Tue Oct 26 17:31:17 2010 yutaUpdateCDSfixed OPLEV stuff and MCL filters

(Joe, Yuta)


 We are currently working on getting rid of "white stuff" in MEDM screens.
 Today, we fixed OPLEV stuff, MCL filters, and time stamps.

What we did:
 1. Plugged in OPLEV cables to ADC2. (See this wiki page for wiring)

 2. Connected ADC2 and OPLEV in Simulink model and fixed MEDM screens for OPLEVs (Attached #1).

 3. Put MCL filters for BS,ITMX,ITMY,PRM,SRM.
  They don't need them, but just for getting rid of "white stuff."
  They are connected to the ground, so the outputs are always 0.

 4. Fixed "TIME_STRING"s in MEDM screens so that they show current time correctly.
  You only need to put text monitor with channel "C1:FEC-DCU_NODE_ID_TIME_STRING" into master files(DEFAULTNAME things) and run generate_master_screens.py.
  It will automatically sets DCU ID correctly!! (Great work, Joe!)
  See this wiki page for more info on making MEDM screens.

 5. Checked OPLEV for MC2 by pointing a laser pointer to QPD. (For MC2, OPLEV is just a transmission beam position monitor)
  Each quadrant looked like they are connected to the right channel numbers.

 - figure out what C1:SUS-NAME_MODE_SW1 does and fix
 - fix Whitening, Dewhitening ON/OFF button in main MEDM screens, so that they switch every channels' filters
 - make a new screen for MC (like the old one C1IOO_ModeCleaner.adl)
 - create a new mark for new MEDM screens

  3789   Tue Oct 26 21:27:02 2010 JenneUpdateCDSfixed OPLEV stuff and MCL filters

Since MC2_TRANS is, in fact, MC2 Transmission, and not an oplev at all (it's not red, and it's not a lever, although it does use a QPD), I propose that the name be changed to something sensical, since calling it an oplev is completely non-sensical.  The name change should happen sooner rather than later, to avoid confusion.


(Joe, Yuta)


 Today, we fixed OPLEV stuff, MCL filters, and time stamps.

What we did:
 5. Checked OPLEV for MC2 by pointing a laser pointer to QPD. (For MC2, OPLEV is just a transmission beam position monitor)
  Each quadrant looked like they are connected to the right channel numbers.


  3794   Wed Oct 27 11:34:40 2010 josephbUpdateCDSModified rc.local for front end machines

What was the problem:

On reboot, c1sus was in a strange state.  The epics IOC was running, along with tpman, but there were no loaded front ends.

People had to manually run sudo insmod c1SYSfe.ko.

What was the cause:

Awhile back Alex had commented out the line in the rc.local file which actually loaded the front end modules (i.e. c1x02fe.ko, c1susfe.ko, etc).  This was while debugging.  He never put it back.

What was the solution:

I uncommented the following line in /diskless/root/etc/rc.local file on the fb machine:


Now on reboot, the c1sus machine should start up all the front ends listed in the rtsystab file, located in /diskless/root/etc/ on the fb machine.


  3796   Wed Oct 27 12:32:53 2010 josephbUpdateCDSfb rebooted to try and fix testpoints


Test points were unavailable last night, even after reboots of c1sus and even restarting the daqd process on the frame builder.


Its unclear at this time.  My guess is flaky fb and mx_stream codes.  At the moment, the daqd often requires several restarts as it segfaults within a minute or two of restarting it.

What we did (aka treating the symptoms):

We rebooted the frame builder machine.  I also added the daqd and nds processes to the inittab.  Now when these die, they will automatically be restarted.

Steps to add to the inittab on fb

0) If not on fb, ssh -X fb

1) cd /etc/

2) sudo vi inittab or sudo emacs init

3) Add a line like: id:runlevels:action:process

The id is a unqiue 2-4 letter and number identifier for the process

Run levels is the run level of linux that it will start at. 345 will cover the normal cases

action is what to do with the process. Respawn makes it run at startup and also restarts it everytime it dies.

process is the command you want to run

See "man inittab" for more details

In this case we added

daq:345:respawn:/opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc > /opt/rtcds/caltech/c1/target/fb/daqd.log

nds:345:respawn:/opt/rtcds/caltech/c1/target/fb/nds pipe > /opt/rtcds/caltech/c1/target/fb/nds.log

4) Save.

5) Run "sudo /sbin/telinit q".  This forces init to rexamine the inittab file

daqd and nds will now automatically restart when they die.

Continuing issues:

When the frame builder dies, the mx_stream processes on the front ends die as well.  These need to be restarted manually at the moment by using "sudo /etc/restart_streams" while on c1sus.

The framebuilder code shouldn't be this flaky.

  3797   Wed Oct 27 15:38:19 2010 josephb, yutaUpdateCDSIO chassis with bad timing was taken back to Downs


The front end timing was not working properly for 2 of the IO chassis.  They were not being synced to the 1 PPS signal. 

This prevented the use of RFM for communication between front ends because time stamps on the transmitted data did not match the cycle on the receiving machine.


We took one of the incorrectly working chassis over to Downs.  Rolf said he would take a look at it tomorrow morning.

Joe will be going over tomorrow morning to talk with Rolf and see what needs to be done to fix it.


  3799   Wed Oct 27 17:06:48 2010 josephbUpdateCDSMoved c1iscey chassis and host interface board to c1ioo


Need a working IO chassis connected to c1ioo in order to bring the MC_L into the digital realm, and then via RFM transmit to the c1sus machine.

Attempted Solution:

Move the c1iscey IO chassis to c1ioo while the c1ioo chassis is at downs.

The c1iscey chassis however doesn't seem to be talking to the c1ioo computer.  I tried changing the host interface card on the c1ioo chassis.  I took out One Stop Systems HIB2-x4-H interface card with serial number 26638 from the c1ioo computer and put in the One Stop Systems HIB2-x4-H with serial number 35242 in from c1iscey into c1ioo.  Still didn't work.

All the lights are red on the interface card on the actual chassis and its cooling fan isn't spinning. 

Using dmesg on c1ioo shows that it does not see any of the ADC/DAC/BO cards.


I'm going to  wait until tomorrow morning when Rolf gets a chance to look at the c1ioo chassis over at Downs to determine the next step.  If we fix the c1ioo chassis, I'm move the c1iscey chassis and its host interface board back to the end.

  3811   Thu Oct 28 16:38:54 2010 josephbUpdateCDSFlaky fb, reverted inittab changes on fb


Yuta reported many of the signals being displayed by dataviewer "fuzzier" than normal.  And diaggui was not working.

Running "diag -i" reported:

Diagnostics configuration:
awg 21 0 822095893 1
awg 36 0 822095908 1
awg 37 0 822095909 1
tp 21 0 822091797 1
tp 36 0 822091812 1
tp 37 0 822091813 1

This seems to be missing an nds type line between the 3 awgs and the 3 tp lines.

The daqd code (the framebuilder) is being especially flaky today, and I'm starting to see new errors.

[Thu Oct 28 16:13:46 2010] Couldn't open full trend frame file

T-972342780-60.gwf' for writing; errno 13
epicsThreadOnceOsd epicsMutexLock failed.
epicsThreadOnceOsd epicsMutexLock failed.
epicsThreadOnceOsd epicsMutexLock failed.
epicsThreadOnceOsd epicsMutexLock failed.
epicsThreadOnceOsd epicsMutexLock failed.
Segmentation fault (core dumped)


[Thu Oct 28 16:17:06 2010] Couldn't open full frame file
`/frames/full/9723/.C-R-972343024-16.gwf' for writing; errno 13
CA client library tcp receive thread terminating due to a C++ exception
FATAL: exception not rethrown
CA client library tcp receive thread terminating due to a C++ exception
CA client library tcp receive thread terminating due to a C++ exception
FATAL: exception not rethrown
cac: tcp send thread received an unexpected exception - disconnecting
Aborted (core dumped)

What was done today that might have affected it:

A new c1ioo chassis from Downs was connected to c1ioo.  I also connected c1ioo to the DAQ network (192.168.114.xxx) which talks to the frame builder.

I started downloading the necessary files to be able to follow Keith's instructions for a standard control / teststand setup in /opt/apps , /opt/rtapps, etc.  However, it has not actually been installed yet.

Yuta added additional OL channels to the DAQ config for being recorded.

Attempted Fixes:

I reverted the inittab changes I made in this elog.  Didn't help.

I disconnected c1ioo from the DAQ network.  Didn't help.

Rebooted the frame builder machine.  Didn't help.

I've sent an e-mail to Alex describing the problem to see if he has any idea where we went wrong.

Yuta may try restoring the old DAQ channel choices and see if that makes a difference.

Current Status:

daqd framebuilder code still won't stay up.  So no channels at the moment.

  3814   Thu Oct 28 21:20:11 2010 yutaUpdateCDSFlaky fb, tried DAQ re-install, but no help

  Unfortunately, fb is flakier than normal. We can't use dataviewer and diaggui now.
  I thought it might be because editting .ini files(list of DAQ channels) in /cvs/cds/rtcds/caltech/c1/chans/daq/ without using GUI was doing something wrong.
  So, I re-installed DAQ, but it didn't help.

What I did:
1. ssh c1sus, went to /opt/rtcds/caltech/c1/core/advLigoRTS/ and ran
  make uninstall-daq-c1SYS
  make install-daq-c1SYS

It didn't help.
More than that, MC suspension damping went wrong. So;

2. Rebooted c1sus machine.
 I restored MC suspension damping by doing this.
 (Similar thing happened Tuesday when we were trying to lock MC)

  Editting .ini DAQ channel list file wasn't wrong. (or, I failed in finding anything wrong right now)


Attempted Fixes:

Yuta may try restoring the old DAQ channel choices and see if that makes a difference.


  3815   Thu Oct 28 23:17:15 2010 yutaSummaryCDS[EMERGENCY] accidentally deleted daqd

Rana showed me that if c1sus machine runs c1mcs stuff(and c1x02 stuff) only, we can use dataviewer without crashing fb.
Also, if we set correct NDS server and port(fb/8088), we can use diaggui on every machine.

During my investigation on what he did, I accidentally deleted daqd......
I am very very sorry.

I don't know if it helps or not, but all I have is the following information:


[What I deleted]
   -rwxr-xr-x 1 controls controls 6583071 Oct  1 11:57 daqd

[help message for daqd existed]
CDS Data Acquisition Server, Frame Builder, version 2.0
California Institute of Technology, LIGO Project
Client communication protocol version 11.4

        daqd [-f <input frame file name>]
        [-c <configuration file (default -- $HOME/.daqdrc)>]

        [-s <frame writer pause usec (default -- 1 sec)>]

This executable compiled on:
        Fri Oct  1 10:33:18 PDT 2010
        Linux fb #7 SMP Fri Sep 24 14:09:53 PDT 2010 x86_64 Dual-Core AMD Opteron(tm) Processor 8220 AuthenticAMD GNU/Linux

Please help me, Joe.

  3821   Fri Oct 29 11:25:15 2010 josephb, yutaSummaryCDS[EMERGENCY] accidentally deleted daqd


Missing daqd file, i.e. the framebuilder executable.


1) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/

2) Look in the Makefile for a likely build suspect.  In this case it was build dc, which stands for data concentrator.

3) So we ran "make dc"

4) Go to the sub-directory build/dc/ and then copy the daqd file there to the /opt/rtcds/caltech/c1/target/fb directory

5) Test it to ensure we're getting channels (Yay!)

Future Safeguards:

Place the new target directory under SVN control.


  3822   Fri Oct 29 11:29:29 2010 josephbUpdateCDSHow I broke the frame builder yesterday


Long before Yuta came along and deleted daqd, I had done something to prevent the framebuilder code from running at all.


Alex pointed out via e-mail that the corresponded to the inability to access certain frame files due to their permissions being only for root. 

Turns out when I had run the code under the inittab, I forgot to make it use controls, instead of root (which is the default).  This later on caused problems for the code when it tried to access those files, resulting in the wierd errors we saw.


Use chown to change the offending frame files back to controls.


Write a proper inittab script which uses "su controls" before running the daqd code.

  3827   Fri Oct 29 16:43:25 2010 josephbUpdateCDSc1ioo now talking to c1sus


c1ioo was not able to talk to c1sus because of timing issues.  This prevented the mode cleaner length signal (MC_L) from getting to c1sus.


The replacement c1ioo chassis from Downs with a more recent revision of the IO backplane works. 

The c1ioo is now talking to c1sus and transmitting a signal.

We connected the cable hanging off the DAQ interface board labeled MC OUT1 to the MC Servo board's output labeled OUT1.

During debugging I modified the c1x02, c1x03, c1mcs and c1ioo codes to print debugging messages.  This was done by modifying the /opt/rtcds/caltech/c1/advLigoRTS/src/fe/commData2.c file.  I have since reverted those changes.


We still need to check that everything is connected properly and that the correct signal is being sent to the MC2 suspension.


  3828   Fri Oct 29 18:37:33 2010 yutaSummaryCDSdaqd and current CDS status

  Before Joe left(~ 1 hour ago), fb was working for a while. But after he left, daqd core dumped.
  This is maybe because we started c1sus and c1rms again for a delay measurement, just before he left.

What I did:
  I restarted IOP(c1x02) and FE models.
  Now it seems OK (we can use dataviewer and diaggui), but daqd reports bunch of errors like;

    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-ITMX_TO_COIL_0_3_INMON", Connecting to:, Ignored: c1susdaq:42367"
    Source File: ../cac.cpp line 1208
    Current Time: Fri Oct 29 2010 18:07:39.132686519

tp node xx invalid  (xx is 38 to 36)

Current CDS status:

MC damp dataviewer diaggui AWG c1ioo c1sus c1iscex RFM Sim.Plant  ...

   Please add other stuff you need.

Below is an example of how the color code works:


  3829   Sat Oct 30 05:27:53 2010 yutaSummaryCDSCDS time delay measurement

  We want to know the time delay of CDS in the IOP scheme.


What I did:
1. Plugged out SCSI cable from ADC card 2 and DAC card 0 on C1SUS machine.
   ADC card 2 is ADC 0
   DAC card 0 is DAC 0

2. Measured tranfer function between ADC and DAC by SR785 and compared with the downsampling filter in IOP with 65534Hz(=4x16384Hz) sampling frequency.

  As ADC_0_0 corresponds to PRM ULSEN input and DAC_0_0 corrsponds to ULCOIL output, we turned all the filters off and set gains to 0 or 1 so that TF between ULSEN to ULCOIL will be ideally 1. (see this wiki page for channel assigns)

  The filter coefficients for the down sampling filter was found in;
  It was named feCoeff4x.

static double feCoeff4x[9] =
        -1.71662585474518,    0.78495484219691,   -1.41346289716898,   0.99893884152400,
        -1.68385964238855,    0.93734519457266,    0.00000127375260,   0.99819981588176};

3. Calculated the time delay dt using the following formula;
  dt = [pm - pc]/f/360deg    (pm: measured phase, pc: calculated phase from feCoeff4x, f: frequency)

4. Measured TF between the SCSI cables to estimate the effect of the cables and others.
  Disconnected SCSI cables from ADC and DAC, and connected A aad B(see setup).
  I measured both when input coupling of SR785 is DC and AC and see what happens.

  [time delay of the CDS]  (left, middle)
    The time delay gets larger with frequency. The time delay seems to be -175 usec at DC.
    However, the gain seems a little different from my expectation(feCoeff4x). So, there are maybe other filters I don't know.
    I neglected TF of upsampling this time.

  [cable and other effect]  (right)
    The effect to the time delay measurement was tiny by a factor of 10^4 to 10^3 (few nsec).
    But the total cable length was about 5 m and assuming signal speed is 0.6c, delay will be about 30 nsec.
    I don't know what's happening.


  - make a model that does not go through IOP and see the delay caused by IOP

By the way:
  fb daqd is still running for hours!
  Every FEs are running(c1sus,rms,mcs).

  3830   Sat Oct 30 14:35:43 2010 KojiSummaryCDSCDS time delay measurement


Neglecting the digital anti-imaging filter makes the discrepancy. You must take into account your digital filter twice.

I attached the slides I made during my visit for March LVC '09. P.5 would be useful.


  [time delay of the CDS]  (left, middle)
    The time delay gets larger with frequency. The time delay seems to be -175 usec at DC.
    However, the gain seems a little different from my expectation(feCoeff4x). So, there are maybe other filters I don't know.
    I neglected TF of upsampling this time.


  3834   Mon Nov 1 11:34:08 2010 josephbUpdateCDSCA.Client.Exception spam fixed


    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-ITMX_TO_COIL_0_3_INMON", Connecting to:, Ignored: c1susdaq:42367"
    Source File: ../cac.cpp line 1208
    Current Time: Fri Oct 29 2010 18:07:39.132686519

The above exception gets thrown for each channel sent to the framebuilder when you start the frame builder.  Its the 192.168.113.xxx (main martian) and 192.168.114.xxx (DAQ) networks sending the same information.  This makes it hard to see real errors when starting the frame builder.


Configure EPICS channel access to only use one network.  This is done by modifying the /etc/bash/bashrc and the /diskless/root/etc/bash/bashrc files with the following two lines:


The first line tells the computers not to automatically search all attached network devices.  The second tells it to use the 192.168.113.xxx network for EPICS channel broadcasts.

  3836   Mon Nov 1 14:53:30 2010 josephb, alex, rolfUpdateCDSAlex updated FB and mx_streams code


The framebuilder was being flaky.  MX_streams would go down, prevent testpoints from working and so forth.


Send Alex up North for a week to fix the code. 

Alex came back and installed updates to the frame builder and the mx_streams code (Myrinet Express over Generic Ethernet Hardware) used by the front ends to talk to the frame builder.  Instead of 1 stream per model, there's now just 1 per front end handling all communications.

Alex did an SVN update and we now have the latest CDS code.

Self restarting codes:

The frame builder code (daqd) and nds pipe have been added to the fb machine's inittab.  Specifically it calls a script called /opt/rtcds/caltech/c1/target/fb/start_daqd.inittab and /opt/rtcds/caltech/c1/target/fb/start_nds.inittab.

The addition to the /etc/inittab file on fb is:



When these codes die they should automatically restart.

Self starting codes at boot up:

The front ends now start the mx_stream script (which lives in /opt/rtcds/caltech/c1/target/fb/ directory) at boot up.  They call it with the approriate command line options for that front end. It can be found in the /etc/rc.local file.

They look like: mx_stream -s "c1x02 c1sus c1mcs c1rms" -d fb:0

As always, the front end codes to be started are defined in the /etc/rtsystab file (or on fb, in the /diskless/root/etc/rtsystab file).

However, if it does go down you would need to restart it manually, although it seems more robust now and doesn't seem to go down every time we restart the frame builder.

All the usual front end IOCs and modules should be started and loaded on boot up as well.


Current CDS status:

MC damp dataviewer diaggui AWG c1ioo c1sus c1iscex RFM Sim.Plant Frame builder  ...
  3837   Mon Nov 1 15:35:41 2010 josephbUpdateCDSFront end USR and CPU times now recorded by DAQ


We have no record of how long the CPUs are taking to perform a cycle's worth of computation


I added the following channels to the various slow DAQ configuration files in /opt/rtcds/caltech/c1/chans/daq/




To restart the daqd code, simply kill the running process.  It should restart automatically.  If it appears not to have started, check the /opt/rtcds/caltech/c1/target/fb/restart.log file and the /opt/rtcds/caltech/c1/target/fb/logs/daqd.log.xxxx files.  If you made a mistake in the DAQ channels and its complaining, fix the error and then restart init on the fb machine by running "sudo /sbin/init q"

ELOG V3.1.3-