[Joe, John]

The nds 2 server is about 50% of the way there.  You can connect to it and get channel lists, but its having issues actually serving the data.  The errors we're getting basically say it can't find the source data for the channel

John had to go get lunch at 2pm, but he said he'd log in remotely later and try to configure it properly.

I added all the _OUT16, _INMON, _EXCMON channels associated with the BS, ITMX, and ITMY channels in SUS_SLOW.ini in the /opt/rtcds/caltech/c1/chans/daq directory.  Similarly, I added the channels associated with MC1, MC2, and MC3 to MCS_SLOW.ini and those associated with PRM and the SRM to RMS_SLOW.ini. 

Lines pointing to these files were then added to the /opt/rtcds/caltech/c1/target/fb/master file and the frame builder restarted.  It took about 4 tries before the frame builder stayed up using ./daqd -c ./daqdrc.

I generated the .ini files with a python script.  Its at /opt/rtcds/caltech/c1/scripts/daqscripts/create_inmon_out16_daq_ini.py. It checks the C0EDCU.ini to find what slow channels already exist, then goes through a medm directory given at the command line looking for all _OUT16, _INMON, and _EXCMON channels, adding them if they aren't already accounted for, and then writing out the new file to the location of your choice.

Why EXCMON? I think that we should modify the filter coefficients so that the decimation filter that makes OUT16 is a 2nd order Butterworth at 1 Hz instead of whatever bogus thing it is now. Then we can delete the EXCMON from the DAQ and add either OUTPUT or OUTMON. That way we will have a low frequency signal (OUT16) and a sort of aliased RMS (OUTMON).

It looks as though we may have two IO chassis with bad timing cards.

Symptoms are as follows:

We can get our front end models writing data and timestamps out on the RFM network.

However, they get rejected on the receiving end because the timestamps don't match up with the receiving front end's timestamp.  Once started, the system is consistently off by the same amount. Stopping the front end module on c1ioo and restarting it, generated a new consistent offset.  Say off by 29,000 cycles in the first case and on restart we might be 11,000 cycles off.  Essentially, on start up, the IOP isn't using the 1PPS signal to determine when to start counting.

We tried swapping the spare IO chassis (intended for the LSC) in ....

# Joe will finish this in 3 days.

# Basically, in conclusion, in a word, we found that c1ioo IO chassis is wrong.

(Koji, Yuta)

Summary:
 The damping servos of the MC suspensions has not been working since yesterday. They had worked on Friday.
 We found that some of the filter coefficients somehow changed from the correct value during dealing with the sampling frequency shift (from 2048Hz to 16384Hz). (see elog #3659)

What we did:
We thought that the problem occurred from the CDS update on Monday. So, we restored them using svn.
 1. Went to /opt/rtcds/caltech/c1/core/advLigoRTS and ran svn update using the revision number 2125.
   svn update -r 2125
 2. Rebuilt all the front ends.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/core/advLigoRTS
   make clean-SYSNAME
   make SYSNAME
   make install-SYSNAME    (where SYSNAME=c1x02,c1sus,c1mcs,c1rms)
 3. Restarted the front end models.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/scripts
   sudo startc1SYS    (where SYS=sus,mcs,rms)
 4. Restarted mx_streams.
   sudo /etc/restart_streams
See this wiki page for the restart procedures.

At this point we judged that the realtime system and "dataviewer" worked fine (by poking some suspensions / by measuring PSDs/TFs of the signals).

However, just restoring the RT system didn't fix the damping servos.

After that, we measured the transfer functions of the servo filters using DTT(diaggui on rosalba).
 5. The filter at MC1_SUSPOS, "3:0.0" should have a cut-off frequency at 3Hz, but it was around 0.3Hz.
 6. We used foton in order to look at the filter bank files (in /cvs/cds/rtcds/caltech/c1/chans). Then we found that the filter design was somehow changed to
   zpk([0],[0.375],2.66666,"n")
  instead of the correct one
   zpk([0],[3],0.333333,"n")

Why the filter designs changed?:
 We suspect that the filter designs were changed because of the replacement of sampling frequency in filter bank files(see elog #3659). We only replaced the lines like
  # SAMPLING SUS_MC3_SUSPOS 2048
 to
  # SAMPLING SUS_MC3_SUSPOS 16384
 and didn't changed the coefficients.
 This may caused a problem when opening the files with foton, and foton changed 3Hz to 0.375Hz (2048/16384) and other coefficients.

Note:
 The damping servo for SIDE has been working for every MCs. POS, PIT, YAW are the ones not working.

Next work:
 - Fix filters for every suspensions.
   There are backup files in /cvs/cds/rtcds/caltech/c1/chans/filter_archive directory, so this should help.
   But I don't think just fixing filters would fix the damping servo because the filter design of MC suspensions were changed this morning and the damping had worked until Friday.

(Rana, Yuta)

Summary:
 The damping servo for MCs has not been working since Monday.
 We already found that some filter coefficients were wrong.(see elog #3742)
 So, we fixed it (just for MCs now).
 But still doesn't work.

What we did:
Fixed the filters for MC suspensions;
 1. In /cvs/cds/rtcds/caltech/c1/chans/ directory, we replaced C1MCS.txt with ./filter_archive/c1mcs/C1MCS_101019_101927.txt.
  This is the archive of the filter bank for MC suspensions, when filter designs were correct(when we hadn't opened it with foton yet).

 2. Deleted all the filter coefficients.
   By using emacs magical C-<space> C-x r k.

 3. Loaded with foton to get correct coefficients.

 4. Reloaded coefficients for C1MCS.

We now have the correct filters, but the damping still doesn't work.

 5. To confirm that the filters are now correct and the model is working correctly, we measured the transfer function between ULSEN input and ULCOIL output and compared with the calculated TF from the filter bank file (Attachment #1) .

 6. To calculate the designed function, we used the following matlab scripts;
    /cvs/cds/caltech/users/rana/mat/utilities/FotonFilter.m        # takes the foton file and returns the TF
    /cvs/cds/caltech/users/rana/mat/utilities/readFilterFile.m    # reads the foton file
    /cvs/cds/caltech/users/yuta/scripts/sentocoil.m                   # calculates the total TF from SEN to COIL

Result:
 As you can see from the attachment, filters seem fine now. (The red line is the measured and the blue line is the calculated function in the plot)
 But the damping (for POS, PIT, YAW) still doesn't work. Why????

Next work:
 - see if coils are pushing correctly
 - push cable connectors further!
 - check input and output filters (whitening, dewhitening)
 - when fixed, use my fully updated QAdjuster.py for adjusting Q for multiple channels automatically!!

(Joe, Yuta)

Background:
 We found that the damping servo for MC suspensions somehow worked when we turned off the 13Hz Chebyshev filters.
 But that does not meet our satisfaction, so we started checking every components.
 First of all is the whitening filters.
 If we turn on a digital whitening filter(WF), corresponding analog WF should turn off, and vice versa.

Reference:
 See DCC #D000210 for the analog circuit of WF. WF has 3Hz zero, 30Hz pole, 100Hz pole. MAX333A bypasses analog WF when supplied +15V(HIGH).

What we did:
 1. Compared the transfer function between MC2_SUSPOS_EXC and MC2_ULSEN_IN1 when digital WF on and off. When digital is on/off, analog should be off/on, so there should be difference but couldn't see.

 2. We went through the simulink model and found 2 mistakes in the logic. One is the conflict with other optics. Even if we turn on/off digital WF of MC2, it didn't switched analog WF of MC2. Two is the additional bit invert (but it turned out to be our misunderstanding).

 3. We (thought we) fixed it, rebuild it, and measured the TF again.(Attachment #1)

[Attached #1]
 The red/blue line is when digital WF is on/off. Blue should be bigger(+10dB @ 10Hz according to (analog) WTF) than red, but it was the opposite.

 4. To confirm that they are doing the opposite, I checked MAX333A input(pin#1,10,11,20) in "SUS PD Whitening Board" at 1X5-1-8B (which is for MC3) and found that switching is opposite. When I turned off/on the digital WF, MAX333A input was +15V/0V. It should be 0V/+15V.

 5. Also, I found that LLSEN digital WF switch switches LRSEN analog WF and vice versa.

[Attached #2]
 Transfer functions between MC2_SUSPOS_EXC and MC2_LRSEN_IN1 with;
   Red: LR digital off, LL digital on
   Blue: LR digital off, LL digital off
   Green: LR digital on, LL digital off
 As you can see, LL switch is the one which switches LR analog WF now.

Conclusion:
 Currently, digital WF on/off corresponds to analog WF on/off.
 Also, LL/LR digital WF switch is LR/LL analog WF switch now.

Next work:
 - fix the simulink models (or wiring)
 - check dewhitening filters

Schematic:
 - whitening
   MC1 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC2 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
   MC3 5 PD outputs -> SUS PD Whitening Board(D000210) -> ... (digital WF=3,100:3)
      D000210 has switches for bypassing analog WT(3,100:3). HIGH to bypass.
 - dewhitening
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,3 UL/UR/LR/LL coils
  (-) ... -> SOS Dewhitening Board(D000316) -> MC1,2,3 SIDE coils
  (SimDW)(InvDW) ... -> LSC Anti-imaging Board(D000186) -> Universal Dewhitening Board(D000183) -> MC2 UL/UR/LR/LL coils
     D000316 has switches for bypassing 28Hz elliptic LPF. HIGH to bypass.
     D000186 is 7570Hz elliptic LPFs.
     D000183 has switches for bypassing dewhitening filter. HIGH to bypass.
 See this wiki page for more comprehensive setup.

(Joe, Yuta)

Summary:
 We found some mistakes in whitening filter switches yesterday(see elog #3748).
 One was the mis-connection between LL/LR and the second was the bit invert.
 So, we fixed it and checked that they are now switching correct by probing MAX333A.

Why we made mistake:
1. LL/LR mis-connection
  Simply, Simulink model was wrong.
  It occurred because UL/UR/LR/LL order is currently arbitrary and confusing.
  For now, we just swap the connection(Attached #1).
  In the future, we should fix all orders corresponding to the actual wiring order(UL > LL > UR > LR).

2. Bit invert
  It occurred from counter-intuitive output of Contec 32 BO(See Attachment #2(taken from this wiki page)).
  If "1," current goes from A to B, so MAX333A input will be 0V, which means analog WF is on.
  If "0," no current, so MAX333A input will be +15V, which means analog WF is off(bypassed).

Next work:
 - There are no digital WF in SDSEN inputs. So, we have to put them.
 - Dewhitening filters are totally messed up now. Switches are wrong, some optics have digital DW but some doesn't, some have 28Hz elliptic LPF.......
     We have to grand unify them.

(Joe, Yuta)

Summary:
 No more discrimination for SIDE!
 We had all SIDE filters in SDSEN, so we splitted into SDSEN, SUSSIDE, and SDCOIL just like other DOFs. (Not SIDECOIL. SDCOIL from now on!)
 Also, there was a confusion in output filters, so we fixed the filter bank(dewhitening and 28HzELP).
 More than that, I found that "13HzCheby" in SUSPOS, SUSPIT, SUSYAW was actually doing ~1.6Hz Chebyshev, so I fixed it. (No wonder we had no damping when turing "13HzCheby" on!)
 Our new MEDM screen is Attachment #1.

Input filter design:
 Every optics, every OSEM has same analog whitening filter.
 So;

Next work:
 - Fix Simulink connection and logic so that analog/digital switching go correctly.
 - Check if the switching are correct for all channels by measuring transfer functions.
 - Currently, we are focusing on MCs, but we have to do the same fixing for other optics, too.
 - More than setting the right filter TF, there are switching settings like Zero History, Ramp, Zero Crossing...... We should investigate that.
 - Actually, TF of analog DW in D000183 doesn't look like the same with digital SimDW. Why??

(Rana, Joe, Yuta)

We now understand that we never succeeded in converting old fiter files to new filter files.

For example, we just changed the sampling rate with coefficients remained and foton confused, or we forgot to rename some of the module names(ULSEN -> SUS_MC1_ULSEN) ......

This is why we sometimes damped and sometimes didn't, depending on filter switches. New filter files has been always wrong.

So, we started to convert them again.
We have to figure out how to convert the files that Foton accepts correctly.

(Joe, Yuta)

Summary:
 This evening, we fixed output filter analog/digital switchings in Simulink logic, but they didn't actually switched filters.
 That was because what we thought BO0 was BO2 and BO2 was BO0.
 We re-wired, re-labeled the cables.
 We checked it by using a LED(see "test configuration" in this wiki page).
 Also, we checked the Simulink connection by probing MAX333A in SOS Dewhitening Board located at 1X4-8-11B, which switches SIDE coils for all optics.

Next work:
 - Currently, FM10 in UL/UR/LR/LL/SDCOIL filters switch analog/digital, but let's make FM9 do the switching.(See schematic in elog #3758)
 - Check all the logic by measuring transfer functions(maybe single frequency measurement is enough. we can use tdsdmd etc to automate).

(Joe, Yuta)

Summary:
 This Monday, MC suspension damping got something wrong.
 We started to check filters and found that digital filters were wrong because of mis-conversion from old filter files to new files.
 We converted the file again, and with mutual understanding between Foton and us, we finally got correct filters(I hope!).

What we did:
 1. Merged filter files in old /cvs/cds/caltech/chans/ directory into C1SUS.txt(BS,ITMX,ITMY), C1RMS.txt(PRM,SRM), C1MCS.txt(MC123).

 2. Rebuilt the RT models in order to get a correct filter file header(they have list of filter modules).

 3. Concatenate the header with filter design part which we got from step1.

 4. Replaced 'N 2048' with 'N 16384'
   It replaces sampling rate of "XXSEN"s.

Basically, step1-4 was the same with what we did last time. We didn't changed the fitler coefficients, so Foton somehow changed the original filter design.
So, this time, we

 5. Deleted coefficients like we did on Tuesday (see elog #3774).

But Foton couldn't read the file correctly this time. Foton seemed to be unbeatable.
Even if we replaced the sampling rate, Foton kept saying 2048! (This is maybe because Foton's default value is 2048Hz. Everytime Foton notice some editting in the file, he destroys everything. He hates editting)
The problems were always associated with the sampling rate, so

 6. Got back to step4 and undo-ed the replacement.

 7. Foton could read it this time, so I changed the sampling rate one by one using Foton GUI.

 8. Checked filters using Foton's Bode Plot.(Not for all, but some that had problem before)

 9. Splitted SDSEN filters to SDSEN, SUSSIDE, and SDCOIL.

 10. Put some missing whitening filters, and 28HzELPs.
   BS, PRM, SRM didn't have any 28HzELP for SDCOIL.
   ITMX and ITMY SDCOIL had SimDW/InvDW which doesn't make sense(SIDEs don't have analog DW). So, I deleted and replaced with 28HzELP.

F2A issue:
 We failed in sending F2A filters to new filter files.
 These are a little bit complicated because TO_COIL_X_X filters were named ULPOS,URPOS etc before.
 Also, MC3 didn't have any F2As, so maybe we should but the same F2As as MC1/2.
 Note that every F2As are different, and TO_COIL matrix have UL,UR,LL,LR order(not same as INMATRIX).
 Also, SRM had f2pv instead of F2A!

Next work:
 - Check whole filters by actually measuring transfer function between SENs and COILs.
 - Damp MC suspentions, and lock MC.
 - Measure openloop TF and compare with the designed.


How do you read a Foton filter file:
 When you open up a Foton filter file, you see filters like this.

################################################################################
### modulename                                                               ###
################################################################################
# SAMPLING modulename samplingrate
# DESIGN   modulename n filterdesign
# DESIGN   modulename n filterdesign
###                                                                          ###
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
                                                                            a1     a2    b1     b2
n: filter number
 0 for FM1, 1 for FM2, ... , 9 for FM10

x: Input Switching setting
 1 Always On
 2 Zero History

y: Output Switching setting
 1 Immediately
 2 Ramp
 3 Input Crossing
 4 Zero Crossing

z: number of filters cascaded.

v: if y=2, (Ramp Time(sec))*(samplingrate)
   if y=3 or 4, Tolerance

w: (Timeout(sec))*(samplingrate)

 Note that v and w are changed when sampling rate is changed.

 Transfer function will be;
  H(1/z)=G*(1+b1/z+b2/z/z)/(1+a1/z+a2/z/z)
  z=exp(s/fs)
 where fs is the sampling frequency.

Reference:
 Kiwamu Izumi: "Notes about Digital Filters," http://tamago.mtk.nao.ac.jp/izumi/green/DigitalFilter.pdf

After replacing filters, MC suspensions damped  last night.

Further measurement next time.

(Joe, Yuta)

 In elog #3708, we showed how to edit all the similar medm screens easiliy.
 But if there are no master files you want to edit in /opt/rtcds/caltech/c1/medm/master/ directory, you can make one.
 Say, you want to edit all the similar medm screens named C1SUS_NAME_XXX.adl.

1. Copy one of C1SUS_NAME_XXX.adl to /opt/rtcds/caltech/c1/medm/master/ directory as C1SUS_DEFAULTNAME_XXX.adl.

2. Go to that directory and
  sed -i 's/NAME/DEFAULTNAME/g' C1SUS_DEFUALTNAME_XXX.adl

3. Add "_XXX.adl" to file_array list in generate_master_screens.py

4. Edit C1SUS_DEFUALTNAME_XXX.adl

5. Run ./generate_master_screens.py

That's all!!

(Joe,Yuta)

Things to do after making a new Simulink model.

1. ssh c1sus, go to /opt/rtcds/caltech/c1/core/advLigoRTS/ and run
 bash ./makestuff.sh c1SYS

 makestuff.sh does;

  make uninstall-daq-$*
  make clean-$*
  make $*
  make install-$*
  make install-daq-$*
  make install-screens-$*
  sed -i 's/RO \(.*SW[12]R.*\)/\1/' /opt/rtcds/caltech/c1/target/$*/$*epics/autoBurt.req

 If you don't need to re-install DAQs or screens, just run line 2,3,4, and 7 and go to step #6.

2. ssh c1sus, go to /opt/rtcds/caltech/c1/scripts/ and run
 sudo ./startc1SYS

 For now, you have to put "1" in "BURT Restore" in GDS screens with in 5-10 secs.

3. Now the DAQ channel numbers are changed. So, go to /cvs/cds/rtcds/caltech/c1/chans/daq/ and run
 ./activateDAQ.py

 activateDAQ will activate the following DAQ channels for every optics with data rate 2048Hz;
  ULSEN_OUT
  URSEN_OUT
  LRSEN_OUT
  LLSEN_OUT
  SDSEN_OUT
  SUSPOS_IN1
  SUSPIT_IN1
  SUSYAW_IN1

4. Restart fb. See this wiki page.
 Basically you what have to do is kill and restart daqd in fb and restart mx_streams in c1sus.

5. DONE! Burt restore if you want.


6. If you don't need to re-install DAQs or screens;
  a. Go to /opt/rtcds/caltech/c1/target/c1SYS/c1SYSepics and run
    sudo rmmod c1SYSfe

  b. Go to /opt/rtcds/caltech/c1/target/c1SYS/bin/ and run
    sudo insmod c1SYSfe.ko

(Joe, Yuta)

Background:
 We are currently working on getting rid of "white stuff" in MEDM screens.
 Today, we fixed OPLEV stuff, MCL filters, and time stamps.

What we did:
 1. Plugged in OPLEV cables to ADC2. (See this wiki page for wiring)

 2. Connected ADC2 and OPLEV in Simulink model and fixed MEDM screens for OPLEVs (Attached #1).

 3. Put MCL filters for BS,ITMX,ITMY,PRM,SRM.
  They don't need them, but just for getting rid of "white stuff."
  They are connected to the ground, so the outputs are always 0.

 4. Fixed "TIME_STRING"s in MEDM screens so that they show current time correctly.
  You only need to put text monitor with channel "C1:FEC-DCU_NODE_ID_TIME_STRING" into master files(DEFAULTNAME things) and run generate_master_screens.py.
  It will automatically sets DCU ID correctly!! (Great work, Joe!)
  See this wiki page for more info on making MEDM screens.

 5. Checked OPLEV for MC2 by pointing a laser pointer to QPD. (For MC2, OPLEV is just a transmission beam position monitor)
  Each quadrant looked like they are connected to the right channel numbers.

Plan:
 - figure out what C1:SUS-NAME_MODE_SW1 does and fix
 - fix Whitening, Dewhitening ON/OFF button in main MEDM screens, so that they switch every channels' filters
 - make a new screen for MC (like the old one C1IOO_ModeCleaner.adl)
 - create a new mark for new MEDM screens

Since MC2_TRANS is, in fact, MC2 Transmission, and not an oplev at all (it's not red, and it's not a lever, although it does use a QPD), I propose that the name be changed to something sensical, since calling it an oplev is completely non-sensical.  The name change should happen sooner rather than later, to avoid confusion.

What was the problem:

On reboot, c1sus was in a strange state.  The epics IOC was running, along with tpman, but there were no loaded front ends.

People had to manually run sudo insmod c1SYSfe.ko.

What was the cause:

Awhile back Alex had commented out the line in the rc.local file which actually loaded the front end modules (i.e. c1x02fe.ko, c1susfe.ko, etc).  This was while debugging.  He never put it back.

What was the solution:

I uncommented the following line in /diskless/root/etc/rc.local file on the fb machine:

/opt/rtcds/caltech/c1/target/$i/scripts/startupC1rt

Now on reboot, the c1sus machine should start up all the front ends listed in the rtsystab file, located in /diskless/root/etc/ on the fb machine.

Problem:

Test points were unavailable last night, even after reboots of c1sus and even restarting the daqd process on the frame builder.

Cause:

Its unclear at this time.  My guess is flaky fb and mx_stream codes.  At the moment, the daqd often requires several restarts as it segfaults within a minute or two of restarting it.

What we did (aka treating the symptoms):

We rebooted the frame builder machine.  I also added the daqd and nds processes to the inittab.  Now when these die, they will automatically be restarted.

Steps to add to the inittab on fb

0) If not on fb, ssh -X fb

1) cd /etc/

2) sudo vi inittab or sudo emacs init

3) Add a line like: id:runlevels:action:process

The id is a unqiue 2-4 letter and number identifier for the process

Run levels is the run level of linux that it will start at. 345 will cover the normal cases

action is what to do with the process. Respawn makes it run at startup and also restarts it everytime it dies.

process is the command you want to run

See "man inittab" for more details

In this case we added

daq:345:respawn:/opt/rtcds/caltech/c1/target/fb/daqd -c /opt/rtcds/caltech/c1/target/fb/daqdrc > /opt/rtcds/caltech/c1/target/fb/daqd.log

nds:345:respawn:/opt/rtcds/caltech/c1/target/fb/nds pipe > /opt/rtcds/caltech/c1/target/fb/nds.log

4) Save.

5) Run "sudo /sbin/telinit q".  This forces init to rexamine the inittab file

daqd and nds will now automatically restart when they die.

Continuing issues:

When the frame builder dies, the mx_stream processes on the front ends die as well.  These need to be restarted manually at the moment by using "sudo /etc/restart_streams" while on c1sus.

The framebuilder code shouldn't be this flaky.

Problem:

The front end timing was not working properly for 2 of the IO chassis.  They were not being synced to the 1 PPS signal. 

This prevented the use of RFM for communication between front ends because time stamps on the transmitted data did not match the cycle on the receiving machine.

Action:

We took one of the incorrectly working chassis over to Downs.  Rolf said he would take a look at it tomorrow morning.

Joe will be going over tomorrow morning to talk with Rolf and see what needs to be done to fix it.

Problem:

Need a working IO chassis connected to c1ioo in order to bring the MC_L into the digital realm, and then via RFM transmit to the c1sus machine.

Attempted Solution:

Move the c1iscey IO chassis to c1ioo while the c1ioo chassis is at downs.

The c1iscey chassis however doesn't seem to be talking to the c1ioo computer.  I tried changing the host interface card on the c1ioo chassis.  I took out One Stop Systems HIB2-x4-H interface card with serial number 26638 from the c1ioo computer and put in the One Stop Systems HIB2-x4-H with serial number 35242 in from c1iscey into c1ioo.  Still didn't work.

All the lights are red on the interface card on the actual chassis and its cooling fan isn't spinning. 

Using dmesg on c1ioo shows that it does not see any of the ADC/DAC/BO cards.

Status:

I'm going to  wait until tomorrow morning when Rolf gets a chance to look at the c1ioo chassis over at Downs to determine the next step.  If we fix the c1ioo chassis, I'm move the c1iscey chassis and its host interface board back to the end.

Problem:

Yuta reported many of the signals being displayed by dataviewer "fuzzier" than normal.  And diaggui was not working.

Running "diag -i" reported:

Diagnostics configuration:
awg 21 0 192.168.113.85 822095893 1 192.168.113.85
awg 36 0 192.168.113.85 822095908 1 192.168.113.85
awg 37 0 192.168.113.85 822095909 1 192.168.113.85
tp 21 0 192.168.113.85 822091797 1 192.168.113.85
tp 36 0 192.168.113.85 822091812 1 192.168.113.85
tp 37 0 192.168.113.85 822091813 1 192.168.113.85

This seems to be missing an nds type line between the 3 awgs and the 3 tp lines.

The daqd code (the framebuilder) is being especially flaky today, and I'm starting to see new errors.

[Thu Oct 28 16:13:46 2010] Couldn't open full trend frame file
`/frames/trend/second/9723/C-

What was done today that might have affected it:

A new c1ioo chassis from Downs was connected to c1ioo.  I also connected c1ioo to the DAQ network (192.168.114.xxx) which talks to the frame builder.

I started downloading the necessary files to be able to follow Keith's instructions for a standard control / teststand setup in /opt/apps , /opt/rtapps, etc.  However, it has not actually been installed yet.

Yuta added additional OL channels to the DAQ config for being recorded.

Attempted Fixes:

I reverted the inittab changes I made in this elog.  Didn't help.

I disconnected c1ioo from the DAQ network.  Didn't help.

Rebooted the frame builder machine.  Didn't help.

I've sent an e-mail to Alex describing the problem to see if he has any idea where we went wrong.

Yuta may try restoring the old DAQ channel choices and see if that makes a difference.

Current Status:

daqd framebuilder code still won't stay up.  So no channels at the moment.

Summary:
  Unfortunately, fb is flakier than normal. We can't use dataviewer and diaggui now.
  I thought it might be because editting .ini files(list of DAQ channels) in /cvs/cds/rtcds/caltech/c1/chans/daq/ without using GUI was doing something wrong.
  So, I re-installed DAQ, but it didn't help.

What I did:
1. ssh c1sus, went to /opt/rtcds/caltech/c1/core/advLigoRTS/ and ran
  make uninstall-daq-c1SYS
  make install-daq-c1SYS

It didn't help.
More than that, MC suspension damping went wrong. So;

2. Rebooted c1sus machine.
 I restored MC suspension damping by doing this.
 (Similar thing happened Tuesday when we were trying to lock MC)

Conclusion:
  Editting .ini DAQ channel list file wasn't wrong. (or, I failed in finding anything wrong right now)

Rana showed me that if c1sus machine runs c1mcs stuff(and c1x02 stuff) only, we can use dataviewer without crashing fb.
Also, if we set correct NDS server and port(fb/8088), we can use diaggui on every machine.


During my investigation on what he did, I accidentally deleted daqd......
I am very very sorry.

I don't know if it helps or not, but all I have is the following information:

[Backup?]
    /opt/rtcds/caltech/c1/target/fb/daqd.25sep10

[What I deleted]
   -rwxr-xr-x 1 controls controls 6583071 Oct  1 11:57 daqd

[help message for daqd existed]
CDS Data Acquisition Server, Frame Builder, version 2.0
California Institute of Technology, LIGO Project
Client communication protocol version 11.4

Usage:
        daqd [-f <input frame file name>]
        [-c <configuration file (default -- $HOME/.daqdrc)>]

        [-s <frame writer pause usec (default -- 1 sec)>]


This executable compiled on:
        Fri Oct  1 10:33:18 PDT 2010
        Linux fb 2.6.34.1 #7 SMP Fri Sep 24 14:09:53 PDT 2010 x86_64 Dual-Core AMD Opteron(tm) Processor 8220 AuthenticAMD GNU/Linux


Please help me, Joe.

Problem:

Missing daqd file, i.e. the framebuilder executable.

Solution:

1) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/

2) Look in the Makefile for a likely build suspect.  In this case it was build dc, which stands for data concentrator.

3) So we ran "make dc"

4) Go to the sub-directory build/dc/ and then copy the daqd file there to the /opt/rtcds/caltech/c1/target/fb directory

5) Test it to ensure we're getting channels (Yay!)

Future Safeguards:

Place the new target directory under SVN control.

Problem:

Long before Yuta came along and deleted daqd, I had done something to prevent the framebuilder code from running at all.

Cause:

Alex pointed out via e-mail that the corresponded to the inability to access certain frame files due to their permissions being only for root. 

Turns out when I had run the code under the inittab, I forgot to make it use controls, instead of root (which is the default).  This later on caused problems for the code when it tried to access those files, resulting in the wierd errors we saw.

Solution:

Use chown to change the offending frame files back to controls.

Future:

Write a proper inittab script which uses "su controls" before running the daqd code.

Problem:

c1ioo was not able to talk to c1sus because of timing issues.  This prevented the mode cleaner length signal (MC_L) from getting to c1sus.

Solution:

The replacement c1ioo chassis from Downs with a more recent revision of the IO backplane works. 

The c1ioo is now talking to c1sus and transmitting a signal.

We connected the cable hanging off the DAQ interface board labeled MC OUT1 to the MC Servo board's output labeled OUT1.

During debugging I modified the c1x02, c1x03, c1mcs and c1ioo codes to print debugging messages.  This was done by modifying the /opt/rtcds/caltech/c1/advLigoRTS/src/fe/commData2.c file.  I have since reverted those changes.

Future:

We still need to check that everything is connected properly and that the correct signal is being sent to the MC2 suspension.

Background:
  Before Joe left(~ 1 hour ago), fb was working for a while. But after he left, daqd core dumped.
  This is maybe because we started c1sus and c1rms again for a delay measurement, just before he left.

What I did:
  I restarted IOP(c1x02) and FE models.
  Now it seems OK (we can use dataviewer and diaggui), but daqd reports bunch of errors like;

CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-ITMX_TO_COIL_0_3_INMON", Connecting to: 192.168.113.85:42367, Ignored: c1susdaq:42367"
    Source File: ../cac.cpp line 1208
    Current Time: Fri Oct 29 2010 18:07:39.132686519
..................................................................

tp node xx invalid  (xx is 38 to 36)

Current CDS status:

   Please add other stuff you need.

Below is an example of how the color code works:

Motivation:
  We want to know the time delay of CDS in the IOP scheme.

Setup:


What I did:
1. Plugged out SCSI cable from ADC card 2 and DAC card 0 on C1SUS machine.
   ADC card 2 is ADC 0
   DAC card 0 is DAC 0

2. Measured tranfer function between ADC and DAC by SR785 and compared with the downsampling filter in IOP with 65534Hz(=4x16384Hz) sampling frequency.

  As ADC_0_0 corresponds to PRM ULSEN input and DAC_0_0 corrsponds to ULCOIL output, we turned all the filters off and set gains to 0 or 1 so that TF between ULSEN to ULCOIL will be ideally 1. (see this wiki page for channel assigns)

  The filter coefficients for the down sampling filter was found in;
    /cvs/cds/rtcds/caltech/c1/core/advLigoRTS/src/fe/controller.c
  It was named feCoeff4x.

static double feCoeff4x[9] =
        {0.014805052402446,
        -1.71662585474518,    0.78495484219691,   -1.41346289716898,   0.99893884152400,
        -1.68385964238855,    0.93734519457266,    0.00000127375260,   0.99819981588176};

3. Calculated the time delay dt using the following formula;
  dt = [pm - pc]/f/360deg    (pm: measured phase, pc: calculated phase from feCoeff4x, f: frequency)

4. Measured TF between the SCSI cables to estimate the effect of the cables and others.
  Disconnected SCSI cables from ADC and DAC, and connected A aad B(see setup).
  I measured both when input coupling of SR785 is DC and AC and see what happens.

Result:
  [time delay of the CDS]  (left, middle)
    The time delay gets larger with frequency. The time delay seems to be -175 usec at DC.
    However, the gain seems a little different from my expectation(feCoeff4x). So, there are maybe other filters I don't know.
    I neglected TF of upsampling this time.

  [cable and other effect]  (right)
    The effect to the time delay measurement was tiny by a factor of 10^4 to 10^3 (few nsec).
    But the total cable length was about 5 m and assuming signal speed is 0.6c, delay will be about 30 nsec.
    I don't know what's happening.



Plan:
  - make a model that does not go through IOP and see the delay caused by IOP

By the way:
  fb daqd is still running for hours!
  Every FEs are running(c1sus,rms,mcs).

Unsatisfactory.

Neglecting the digital anti-imaging filter makes the discrepancy. You must take into account your digital filter twice.

I attached the slides I made during my visit for March LVC '09. P.5 would be useful.

Problem:

CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-ITMX_TO_COIL_0_3_INMON", Connecting to: 192.168.113.85:42367, Ignored: c1susdaq:42367"
    Source File: ../cac.cpp line 1208
    Current Time: Fri Oct 29 2010 18:07:39.132686519

The above exception gets thrown for each channel sent to the framebuilder when you start the frame builder.  Its the 192.168.113.xxx (main martian) and 192.168.114.xxx (DAQ) networks sending the same information.  This makes it hard to see real errors when starting the frame builder.

Solution:

Configure EPICS channel access to only use one network.  This is done by modifying the /etc/bash/bashrc and the /diskless/root/etc/bash/bashrc files with the following two lines:

export EPICS_CA_AUTO_ADDR_LIST=NO
export EPICS_CA_ADDR_LIST="192.168.113.255"

The first line tells the computers not to automatically search all attached network devices.  The second tells it to use the 192.168.113.xxx network for EPICS channel broadcasts.

Problem:

The framebuilder was being flaky.  MX_streams would go down, prevent testpoints from working and so forth.

Solution:

Send Alex up North for a week to fix the code. 

Alex came back and installed updates to the frame builder and the mx_streams code (Myrinet Express over Generic Ethernet Hardware) used by the front ends to talk to the frame builder.  Instead of 1 stream per model, there's now just 1 per front end handling all communications.

Alex did an SVN update and we now have the latest CDS code.

Self restarting codes:

The frame builder code (daqd) and nds pipe have been added to the fb machine's inittab.  Specifically it calls a script called /opt/rtcds/caltech/c1/target/fb/start_daqd.inittab and /opt/rtcds/caltech/c1/target/fb/start_nds.inittab.

The addition to the /etc/inittab file on fb is:

daq:345:respawn:/opt/rtcds/caltech/c1/target/fb/start_daqd.inittab 

nds:345:respawn:/opt/rtcds/caltech/c1/target/fb/start_nds.inittab

When these codes die they should automatically restart.

Self starting codes at boot up:

The front ends now start the mx_stream script (which lives in /opt/rtcds/caltech/c1/target/fb/ directory) at boot up.  They call it with the approriate command line options for that front end. It can be found in the /etc/rc.local file.

They look like: mx_stream -s "c1x02 c1sus c1mcs c1rms" -d fb:0

As always, the front end codes to be started are defined in the /etc/rtsystab file (or on fb, in the /diskless/root/etc/rtsystab file).

However, if it does go down you would need to restart it manually, although it seems more robust now and doesn't seem to go down every time we restart the frame builder.

All the usual front end IOCs and modules should be started and loaded on boot up as well.

Current CDS status:

Problem:

We have no record of how long the CPUs are taking to perform a cycle's worth of computation

Solution:

I added the following channels to the various slow DAQ configuration files in /opt/rtcds/caltech/c1/chans/daq/

IOO_SLOW.ini:[C1:FEC-34_USR_TIME]
IOO_SLOW.ini:[C1:FEC-34_CPU_METER]
IOP_SLOW.ini:[C1:FEC-20_USR_TIME]
IOP_SLOW.ini:[C1:FEC-20_CPU_METER]
IOP_SLOW.ini:[C1:FEC-33_USR_TIME]
IOP_SLOW.ini:[C1:FEC-33_CPU_METER]
MCS_SLOW.ini:[C1:FEC-36_USR_TIME]
MCS_SLOW.ini:[C1:FEC-36_CPU_METER]
RMS_SLOW.ini:[C1:FEC-37_USR_TIME]
RMS_SLOW.ini:[C1:FEC-37_CPU_METER]
SUS_SLOW.ini:[C1:FEC-21_USR_TIME]
SUS_SLOW.ini:[C1:FEC-21_CPU_METER]

Notes:

To restart the daqd code, simply kill the running process.  It should restart automatically.  If it appears not to have started, check the /opt/rtcds/caltech/c1/target/fb/restart.log file and the /opt/rtcds/caltech/c1/target/fb/logs/daqd.log.xxxx files.  If you made a mistake in the DAQ channels and its complaining, fix the error and then restart init on the fb machine by running "sudo /sbin/init q"

Background:
  I measured CDS time delay last week, but because of my lack of understanding the system, it was incorrect.
  IOP has an anti-aliasing filter before downsampling from 64kHz(65536Hz) to 16kHz(16384Hz) and also has an anti-imaging filter before upsampling from 16kHz to 64kHz.
  So, I should have take feCoeff4x into account twice.


Result:
  TF agreed well with 2-time feCoeff4x and CDS time delay was -123.5 usec.



Plan:
 - make AWG(, diaggui TF measurement, tdssine) work
 - check input/output filter switching (using tdssine & tdsdmd)
 - measure openloop TF of MC suspension damping
 - divide it with my expectation and see if there are any filters I don't know

Um, Beautiful.

Actually, 123.5usec is almost exactly twice of 1/16384Hz.
Because of the loop, we have 1/16384Hz delay. I wonder where we do have the delay.

In order to understand the behaviour of the system can I ask you to test the following things?

1) What are the delay without IOPs with f_sampl of 16k, 32k, 64k?

2) What are the delay with IOP with f_sampl of 32k, 64k?

Frame builder died again!!

Background:
  We want to do angle to length measurement to optimize the beam position and increase visibility of MC locking.
  In order to do A2L measurement, we need excitation point, but AWG is currently not working.
  The better way is to use LOCKIN stuff like we had for OMC and put it to C1IOO WFS.
  A software oscillator in LOCKIN shakes the suspension, and demodulate the length signal.
  We can choose whatever DOF to shake, whatever signal to demodulate. It would be useful not just for A2L.

What I did:
  I started to put C1IOO WFS signal into C1SUS MC suspension RT model, but after compiling new c1mcs, fb crashed.
  Looks like daqd and mx_streams are running, but DAQ is not working(red).
  I don't know how to restart in a new way!

Background:
  For input filter, we have analog whitening filter and also digital whitening filter. They have the same TF and when analog one is off, digital one should be on and vice versa.
  I made a python script that checks the switching automatically.

Method:
  Excite the suspension in a single frequency and see sensor inputs(XXSEN_IN1).
  Calculate the magnitude in the excitation frequency and compare it when digital whitening is off and on.
  When digital whitening is off, analog should be on, so sensor inputs should gone though the analog filter. That means the signal is multiplied by the TF of that filter, which makes the difference.

  We currently don't have excitation and I thought I have to wait, but instead of putting some extra excitation, I found that 60Hz line noise is useful.

Script:
  The script is /cvs/cds/caltech/users/yuta/scripts/WDWchecker.py
  For every sensor input, it;
    0. Stores current filter switching(XXSEN_SW1R)
    1. turns OFF the digital filter(FM1, using ezcaswitch)
    2. tdsdmd XXSEN_IN1 in 60Hz
    3. turns ON the digital filter
    4. tdsdmd XXSEN_IN1 in 60Hz
    5. divides mag(2.) by mag(4.) and calculate the analog filter gain in 60Hz
    6. Restores the filter switching in the state before the checking

Result:
  The results are;

C1:SUS-BS_ULSEN_IN1: 22.2 dB
C1:SUS-BS_URSEN_IN1: 18.7 dB
C1:SUS-BS_LRSEN_IN1: 22.7 dB
C1:SUS-BS_LLSEN_IN1: 16.0 dB
C1:SUS-BS_SDSEN_IN1: 21.5 dB
C1:SUS-ITMX_ULSEN_IN1: 16.9 dB
C1:SUS-ITMX_URSEN_IN1: 16.3 dB
C1:SUS-ITMX_LRSEN_IN1: 17.5 dB
C1:SUS-ITMX_LLSEN_IN1: 17.1 dB
C1:SUS-ITMX_SDSEN_IN1: 6.2 dB
C1:SUS-ITMY_ULSEN_IN1: 15.5 dB
C1:SUS-ITMY_URSEN_IN1: 16.5 dB
C1:SUS-ITMY_LRSEN_IN1: 17.4 dB
C1:SUS-ITMY_LLSEN_IN1: 16.3 dB
C1:SUS-ITMY_SDSEN_IN1: 18.0 dB
C1:SUS-PRM_ULSEN_IN1: 0.1 dB
C1:SUS-PRM_URSEN_IN1: 10.3 dB
C1:SUS-PRM_LRSEN_IN1: 13.1 dB
C1:SUS-PRM_LLSEN_IN1: -32.3 dB
C1:SUS-PRM_SDSEN_IN1: 14.6 dB
C1:SUS-SRM_ULSEN_IN1: 17.3 dB
C1:SUS-SRM_URSEN_IN1: 13.5 dB
C1:SUS-SRM_LRSEN_IN1: 1.6 dB
C1:SUS-SRM_LLSEN_IN1: 16.7 dB
C1:SUS-SRM_SDSEN_IN1: 18.3 dB
C1:SUS-MC1_ULSEN_IN1: 17.0 dB
C1:SUS-MC1_URSEN_IN1: 18.6 dB
C1:SUS-MC1_LRSEN_IN1: 14.9 dB
C1:SUS-MC1_LLSEN_IN1: 27.0 dB
C1:SUS-MC1_SDSEN_IN1: 16.6 dB
C1:SUS-MC2_ULSEN_IN1: 19.8 dB
C1:SUS-MC2_URSEN_IN1: 14.0 dB
C1:SUS-MC2_LRSEN_IN1: 20.8 dB
C1:SUS-MC2_LLSEN_IN1: 16.1 dB
C1:SUS-MC2_SDSEN_IN1: 17.3 dB
C1:SUS-MC3_ULSEN_IN1: 15.5 dB
C1:SUS-MC3_URSEN_IN1: 17.3 dB
C1:SUS-MC3_LRSEN_IN1: 18.2 dB
C1:SUS-MC3_LLSEN_IN1: 18.7 dB
C1:SUS-MC3_SDSEN_IN1: 16.8 dB

  Whitening filter has 18dB gain at 60Hz. (It's 3Hz pole, 30Hz zero, 100Hz zero and 0dB at DC)
  So, from the result, at least MC suspensions look like they have correct switching.
  But some channels doesn't look ok.
  We have to check those.

Plan:
   - check ITMX_SDSEN, PRM_ULSEN, PRM_LLSEN, SRM_LRSEN input filters
   - check the script and see if the script can really check. maybe the script needs some adjustments (# of averaging, multiple frequency, ......)
   - make AWG(, tdssine) work
   - check output hardware filter

By the way:
  fb is back. I don't know why. With help from Joe, I just compiled c1mcs again and again changing number of RFM channels.

Problem:

Diagnostic test tools was starting with errors.

Cause:

After the reboot of the frame builder machine yesterday by Alex, the diagconfd daemon was not getting started by xinetd.  There was a sequence error in the startup where xinetd was being called before mounting drives from linux1.

Important Note:

If you do not see the "nds" line you would not have diagnostic tests enabled in the DTT:

[controls@rosalba apps]$ diag -i | grep nds
nds * * 192.168.113.202 8088 * 192.168.113.202

Solution:

Alex changed /etc/xinetd.d/diagconfd file to point to /opt/apps/gds/bin/diagconfd instead of /opt/apps/bin/diagconf.  He also ensured xinetd started after mounting from linux1.

Alex's Suggestion:
My feeling is we should get rid of this feature and have an NDS address
entry box in the "Online" tab in the DTT with the default "nds". I
mentioned this to Jim Batch and he greed with me, so maybe he is going to
implement this. So maybe you guys want to request the same thing too, send
the request to Rolf and Jim, so we can have the last demon exercised.

Problem:

Each RFM memory location which needs to be read by a front end model slows the model significantly.

With no RFM memory locations to be read (replaced with grounds), the c1mcs model runs around 25 microseconds per cycle.

With 1 RFM memory location (MC_L), it runs around 29-33 microseconds.

With 3 RFM memory locations (MC_L, MC1_PIT, MC1_YAW), it runs around 45 microseconds.

With 7 RFM memory locations, the code generally doesn't run at all, going past the 62 microsecond maximum required to be able to keep up with the 16 kHz sample rate.

Last night Yuta somehow got it running with 7 RFM memory locations, but in that case, all the odd numbered RFM channels (1,3,5 as counted by the ipc file) did not work.  It was running at around 55 microseconds in that case.

The c1ioo code which is writing the data to the RFM card is experiencing no such slow down.

Current CDS status:

In order to have the c1mcs model run, we're running with only 3 RFM channels between c1ioo and c1mcs at the moment.  This leaves the model at around 45 microseconds, and at least lets us damp.

Alex and I still need to track down why the RFM read calls are taking so much time to execute.

Summary:
  Last night, I found that some of the input channels have wrong hardware filter switching(see elog #3842).
  So, to check the whitening board(D000210), I swapped the one with ok switching and bad switching.
  During the checking, I somehow broke the board.
  I fixed it, and now the status is the same as last night (or, at least look like the same).

What I did:
  1. Switching for SRM_LRSEN looked bad and every input channel for MC3 looked OK.
     So, I unplugged the whitening board for SRM (1X5-1-5B) and plugged it into MC3's place(1X5-1-8B).

  2. Ran WDWchecker.py for MC3. The switching seemed OK for every input channel, which means the whitening board was not the wrong one.

  3. Swapped back the whitening board as it was.

  4. Found MC3_ULSEN_OUT and MC3_LLSEN_OUT was keep showing negative value(they should be positive).

  5. Check the board and found that one of LT1125 for UL/LL was wrong (broken virtual ground).

  6. Replaced LT1125 and put the board back to 1X5-1-8B.

  7. Checked the board with WDWchecker.py and dataviewer 5-hour minute trend.
      The input signal came back to normal value(Attachment #1), MC3 damping working, input filter switching seems working

Result:
  The whitening board seems OK.
  The wrong one is either wiring or RT model. Or, the checking script.

When swapping Eurocard boards, it is safest to first turn down the power supplies and then do the swapping.

Otherwise, it is sometimes the case that people plug in the board slowly and/or asymmetrically. This can cause the opamp to see one of the power supply rails without the other (e.g. +15 but not -15).

This can destroy some opamps. The true danger is that there may be damage to the board which you do not notice for several months, thereby leaving a timebomb for the next person.

Don't be an electronics terrorist!

Problem:

Each RFM read in the c1mcs model is adding ~7 microseconds to the cycle time.  Adding too many pushes it over the 62 microsecond limit.

RFM writes do not have this problem.

Temporary Solution:

The fastest fix was to create a new front end model, called c1rfm, which does nothing but read in the MC1, MC2, MC3 PIT and YAW signals from the c1ioo machine, and then passes them along to the c1mcs model via shared memory, which is fast.

This means the data being sent is 2 cycles slow, one to go over the RFM, and one to go over shared memory.  It is running at 16384 cycles/second, so it shouldn't have much impact at the frequencies we use those channels for.

MC_L is still being sent directly to the c1mcs front end code via RFM.

Current Status:

The c1mcs model is running at  30-33 microseconds for CPU time.

The c1rfm model is running at 45-47 microseconds for CPU time.

All 7 channels, MC_L, MC1_PIT, MC1_YAW, MC2_PIT, MC2_YAW, MC3_PIT, MC3_YAW are responding.

The c1rfm model was added to the /diskless/root/etc/rtsystab file on the fb machine so that it automatically starts on the reboot of c1sus.

The USR and CPU time channels for c1rfm were added to the MCS_SLOW.ini file in /opt/rtcds/caltech/c1/chans/daq/ so that the framebuilder records them, namely:

[C1:FEC-38_USR_TIME]
[C1:FEC-38_CPU_METER]

The framebuilder was restarted to take these new channels into account.

Future:

Finish implementing and debugging the "round robin" RFM reader so as to not require a seperate model to be doing RFM reads in parallel.

Look into improving read speed by either merging timestamps and data into a single  or reading time stamp once every tenth or hundreth cycle, although this at best provides a factor of 2 improvement.

Check to see if RFM card being on the IO chassis or directly in the computer chassis makes a difference.

Get Alex and Rolf to improve RFM read speed.

Problem:

RFM reads are slow.  Rolf has said it should take 2-3 microseconds per read. 

c1sus is taking about 7 microseconds per read, twice as slow as Rolf's claim.

Hypothesis:

The RFM card is in the IO chassis, and is sharing the PCIe bus with 4 ADC cards, 3 DAC cards, 4 BO cards, and a BIO card.  Its possible this crowded bus is causing the reads to take even longer.

Test Results:

Compare read times between the c1sus computer, which has its RFM card in the IO chassis, to c1ioo, which has its RFM card in the computer.

c1ioo:

No RFM reads: 8 microseconds

3 RFM reads: 20 microseconds (~4 per read)

6 RFM reads: 32 microseconds (~4 per read)

c1sus:

No RFM read: 25 microseconds (bigger model)

1 RFM read: 33 microseconds (~8 per read)

3 RFM read: 45 microseconds (~7 per read)

6 RFM read: Over 62 microseconds, doesn't run.

Conclusion:

It looks like moving the RFM card may help by about a factor of 2 in read speed, although its still not quite what Alex and Rolf claim it should be.

The c1mcs and c1ioo models have been reverted to their normal operations.

(Joe, Yuta)

Summary:
  LOCKIN(consists of oscillator and demodulator) is needed for A2L measurement.
  So, we put LOCKIN to c1ioo model, whose outputs goes to c1mcs ASC.
  After that, I checked the functionality of LOCKIN by directly connecting DAC output for a coil to ADC input for MCL with BNC cable.

What we did:
[Putting LOCKING to c1ioo model]
1. Copied Simulink LOCKIN stuff(cdsOsc, Product, cdsPhase ...) from /cvs/cds/caltech/cds/rward-advLigo/src/epics/simLink/omc.mdl and put it into c1ioo model.

2. Copied MEDM screen file /cvs/cds/caltech/medm/c1/omc/C1OMC_LSC_LOCKIN.adl and modified it for our use.

[Checking LOCKIN]
3. Disconnected MC2_ULCOIL input to SOS Coil driver at 1X4-1-6A and checked the signal from software oscillator at c1ioo is coming.

4. Disconnected the cable labeled "MC OUT1" at 1X2 (which is MCL signal to ADC) and put MC2_ULCOIL output directly using long BNC cable.

5. Checked the functionality of LOCKIN by StripTool.
   The cable wiring did not conflict with my expectation.
   Software mixer is working.(frequency is doubled. X_SIN has offset and X_COS doesn't)

6. Put the cables back.

Result:
  Thanks to c1rfm, c1ioo and c1sus are talking without ADC timeout.
  Also, LOCKIN is working fine.

Feature Requested:

Have the CPU_meter change change color at various alarm levels.  These alarm levels have been set at 2/3 maximum for Minor alarm (yellow) and 9/10 maximum for Major alarm (red).

Implementation:

Rather than hand code each EPICS .db file to add the alarm files each time we rebuild the front ends, I decided to modify it at the source (since it strikes me as a generally useful alarm level for all front end codes).

First, I modified the feCodeGen.pl script.

I changed

print EPICS "OUTVARIABLE FEC\_$dcuId\_CPU_METER epicsOutput.cpuMeter int ai 0 field(HOPR,\"$rate\") field(LOPR,\"0\")\n";

to

print EPICS "OUTVARIABLE FEC\_$dcuId\_CPU_METER epicsOutput.cpuMeter int ai 0 field(HOPR,\"$rate\") field(LOPR,\"0\") field(HIGH,\"$two_thirds_rate\") field(HSV,\"MINOR\") field(HIHI,\"$     nine_tenths_rate\") field(HHSV,\"MAJOR\")\n";

I added the following two lines just before it as well:

$two_thirds_rate = int($rate * 2 / 3);

$nine_tenths_rate = int($rate * 9 / 10);

However, only the first four fields were actually added to the database file.  Apparently fmseq.pl, which populated the database, was hard coded to only handle up to 4 fields.

I modified the fmseq.pl script in /opt/rtcds/caltech/c1/core/advLigoRTS/src/epics/util/ so as to be able to handle up to 6 field values when writing EPICS .db files.

This change was accomplished by simply changing the following line

($junk, $v_name, $v_var, $v_type, $ve_type, $v_init, $v_efield1, $v_efield2, $v_efield3, $v_efield4 ) = split(/\s+/, $_);

to 

($junk, $v_name, $v_var, $v_type, $ve_type, $v_init, $v_efield1, $v_efield2, $v_efield3, $v_efield4, $v_efield5, $v_efield6 ) = split(/\s+/, $_);

everywhere it occurred.  There were something like 10 instances of it. Also, I added the two lines

        $vardb .= "    $v_efield5\n";
        $vardb .= "    $v_efield6\n";

after each set of

         $vardb .= "    $v_efield1\n";
         $vardb .= "    $v_efield2\n";
         $vardb .= "    $v_efield3\n";
         $vardb .= "    $v_efield4\n";

Lastly, I modified the CPU_METER bar graph on the C1SUS_DEFAULTNAME.adl screen (located in /opt/rtcds/caltech/c1/medm/master/) to use alarm levels, and then ran generate_master_screens.py.

Problem:

We can't access any of the c1ioo computer testpoints.  Dataviewer and DTT both fail to read any data from them.

According to diag -l (when run on Rosalba in /opt/apps), the testpoints are not being set.  Also at some point during the day when debugging, we also somehow messed up all the front end connections to the framebuilder.

Errors reported by dataviewer:

Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
read(); errno=0
Server error 6532: invalid channel name
Server error 16080: unknown error
datasrv: DataWriteRealtime failed: daq_send: Illegal seek

Error reported by daqd.log:

[Thu Nov  4 13:29:35 2010] About to request `C1:IOO-MC1_PIT_IN1' 10022
on node 34
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10022; tp[1]=32531

[Thu Nov  4 13:29:35 2010] About to request `C1:SUS-MC2_SUSPOS_IN1'
10094 on node 36
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10094; tp[1]=32531

[Thu Nov  4 13:29:38 2010] ETIMEDOUT: test point `C1:IOO-MC1_PIT_IN1'
(tp_num=10022) was not set by the test point manager; request failed
[Thu Nov  4 13:29:38 2010] About to clear `C1:IOO-MC1_PIT_IN1' 10022 on node 34

Attempted Fixes:

Remove all daq related files: /opt/rtcds/caltech/c1/target/gds/param/tpchn_c1ioo.par and /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini.  Rebuilt the front end code.

Double checked ethernet connection between c1ioo and the daq router and fb. 

Confirmed open mx was running on c1ioo. Confirmed awgtpman was running (although at one point I did find duplicate awgtpman running for c1x03, the IOP associated with c1ioo).

Rebooted the c1ioo machine. Confirmed all necessary codes came back.

Restarted the daqd process several times on the fb machine.

Current Status:

Framebuilder is running, and c1sus testpoints are available.  c1ioo test points are not.  Waiting to hear back from Alex on possible ideas.

Summary:
  I wrote a python script for A2L measurement.
 Currently it is really primitive, but I tested the basic functionality of the script.
 We already have A2L script(at /cvs/cds/rtcds/caltech/c1/scripts/A2L) that uses ezlockin, but python is more stable and easy to read.

A2L measurement method:
  1. Dither a optic using software oscillator in LOCKIN and demodulate the length signal by that frequency.
  2. Change coil output gains to change the pivot of the dithering and do step 1.
  3. Coil output gain set that gives the smallest demodulated magnitude tells you where the current beam spot is.

  Say you are dithering the optic in PIT and changing the coil gains keeping UL=UR and LL=LL.
  If the coil gain set UL=UR=1.01, LL=LR=-0.99 gives you demodulated magnitude 0, that means the current beam spot is 1% upper than the center, compared to 1/2 of UL-LL length.
  You do the same thing for YAW to find horizontal position of the beam.

Description of the script:
  Currently, the script lives at /cvs/cds/caltech/users/yuta/scripts/A2L.py
  If you run;
     ./A2L.py MC1 PIT
  it gives you vertical position of the beam at MC1.

  It changes the TO_COIL matrix gain by "DELTAGAINS", turns on the oscillator, and get X_SIN, X_COS from C1IOO_LOCKIN.
  Plots DELTAGAINS vs X_SIN/X_COS and fit them by y=a+bx+cx^2.(Ideally, c=0)
  Rotates (X_SIN, X_COS) vectors to get I-phase and Q-phase.
    (I,Q)=R*(X_SIN,X_COS)
  Rotation angle is given by;
    rot=arctan(b(X_COS)/b(X_SIN))
  which gives Q 0 slope(Ideally, Q=0).
  x-intercept of DELTAGAINS vs I plot gives the beam position.

Checking the script:
  1. I used the same setup when I checked LOCKIN(see elog #3857). C1:SUS-MC2_ULCOIL output goes directly to C1:IOO-LOCKIN_SIG input.

  2. Set oscillator frequency to 18.13Hz, put 18.13Hz band-pass filter to C1:IOO-LOCKIN_SIG filter module, and put 1Hz low-pass filter to C1:IOO-LOCKIN_X_SIN/X_COS filter modules.
        Drive frequency 18.13Hz is same as the previous script(/cvs/cds/rtcds/caltech/c1/scripts/A2L/A2L_MC2).

  3. Ran the script. Checked that Q~0 and rot=-35deg.

  4. Put phase shifting filter to C1:IOO-LOCKIN_SIG filter module and checked Q~0 and rotation angle.
     fitler rot(deg)
     w/o    -35
     +90deg  45
     -90deg  56
     -45deg -80

  5. Put some noise in C1:SUS-MC2_ULCOIL by adding SUSPOS feedback signal and ran the script.(Attachment #1)
      During the measurement, the damping servo was off, so SUSPOS feedback signal can be treated as noise.

Conclusion:
  The result from the test measurement seems reasonable.
  I think I can apply it to the real measurement, if MCL signal is not so noisy.[status: yellow]

Plan:
  - add calculating coherence procedure, averaging procedure to the script
  - add setting checking procedure to the script
  - apply it to real A2L measurement

Bay the way:
  Computers in the control room is being so slow (rossa, allegra, op440m, rosalba). I don't know why.

Problem:

The 40m computers were responding sluggishly yesterday, to the point of being unusable.

Cause:

The mx_stream code running on c1iscex (the X end suspension control computer) went crazy for some reason.  It was constantly writing to a log file in /cvs/cds/rtcds/caltech/c1/target/fb/192.168.113.80.log.  In the past 24 hours this file had grown to approximately 1 Tb in size. The computer had been turned back on yesterday after having reconnected its IO chassis, which had been moved around last week for testing purposes - specifically plugging the c1ioo IO chassis in to it to confirm it had timing problems.

Current Status:

The mx_stream code was killed on c1iscex and the 1 Tb file removed.

Computers are now more usable.

We still need to investigate exactly what caused the code to start writing to the log file non-stop.

Update Edit:

Alex believes this was due to a missing entry in the /diskless/root/etc/hosts file on the fb machine.  It didn't list the IP and hostname for the c1iscex machine.  I have now added it.  c1iscex had been added to the /etc/dhcp/dhcpd.conf file on fb, which is why it was able to boot at all in the first place.  With the addition of the automatic start up of mx_streams in the past week by Alex, the code started, but without the correct ip address in the hosts file, it was getting confused about where it was running and constantly writing errors.

Future:

When adding a new FE machine, add its IP address and its hostname to the /diskless/root/etc/hosts file on the fb machine.