We had fire alarm tests  and evacuation drills at 1:30pm yesterday. All flashers and horns are functioning unbearably loud and bright including clean assembly room.

I think rana did some more changes to this workstation to make it useful for commissioning activities - but the MEDM screens were still white blanks. The problem was that the firewalld wasn't disabled (last two steps of the KThorne setup wiki). I disabled it. Now donatella can run MEDM, ndscope and StripTool. DTT doesn't work to get online data because of a "Synchronization Error", I'm not bothering with this for now. I think Kruthi successfully demonstrated the fetching of offline data with DTT.

Restocked First Aid Kits Location:

Main entrance, room 100

Drill press - above N2 cylinders, room 103

Control room, next to fire extinguisher, room  102

Vertex-north wall, IFO room 104

ETMY - right on ends light switches, IFO room 104_ east end

ETMX - on vertical I-beam of crane, IFO room 104_south end

Behind 1X3 Rack, on south wall - under instrument breakers panel PC-1, IFO room 104

Last thing remaining to be fixed from 2013 Safety Audit:

replace book shelf with 83" height

In order to verify our theory about coherence corruption in linear systems due to the line

if((new_hist < 1e-20) && (new_hist > -1e-20)) new_hist = new_hist<0 ? -1e-20: 1e-20;  

in the /opt/rtcds/caltech/c1/core/release/src/include/drv/fm10Gen.c in the iir_filter function I've changed -20 to other numbers and watched at the coherence input and output of the digital filter cheby1("LowPass", 3, 0.1, 0.5)cheby1("LowPass", 6, 1, 1.5). The sampling rate was 2K. The frequency responce of the filter presented in this figure.

The next plot shows psd and coherence of the signal for different numbers in the if-statement line : 1e-20 , 1e-25, 1e-100.

We can see that for present value coherence between input and output signals is small even for low frequencies. The psd of the output signal is also corrupted because at low frequencies it should have the same psd as input signal. For 1e-25 and 1e-100 we can see that coherence is close to 1 at low frequencies so if-statement does not work and we have a first order transition from bad to good filter performance with discontinious jump of coherence.

However, for 1e-25 and 1e-100 data is still corrupted by the round-off error. Lack of coherence for high frequencies can be explained by the fact that dtt tools use single precision for data analysis and output is too small to plot a right coherence. But the coherence is also not precisely 1 for low frequencies. Actually, it is 0.99 for this aggresive filter. We use double precision in the real-time code but still for such kinds of filters round-off error is present. As wrote Daniel Sigg for Cheby filter:  "You need a lot more digits than you may naively suspect. In the 8th order example, the output of each SOS is amplified by ~10⁶. This regardless of the fact that the coefficients are all of order 1. If you require a level of 10^-3 attenuation in the stop band, you would have lost 9 digits already. Then, add the fact that you have to do of order 10⁴ subtractions to get from 16kHz to 12Hz, loosing another ~2 digits. On top, the high Q section is probably 10 worse than the others and you lost 12 digits. In a real example this may stack up even worse."

Next we need to figure out what effects does round-off error introduce in the performance of the interferometer.

 Pumpdown 75 - Maglev - day 12

Precondition: 66 days at atm, installed TIP-TILTs with coils that  replaced PZT-Jena input steering

need to vary start/stop times in fit to test for systematics

Pasadena got 0.2" of  rain on Saturday. Temperatures were high with high humidity since than.The ants were back in the Control room east side benches.

We have started using TERRO Liquid Ant Baits in January 2015   This worked very well to this point.

Tree new packages were opened  yesterday and the ants are gone.

We can conclude that these baits must be replaced after 6 mounts.

The liquid baits contains BORAX  and it is safe.

 [Joe, Kiwamu]

We found that the vertex watchdogs were not correctly running.

After I powercycled c1susaux, the problem was fixed successfully.

The symptom: the watchdogs didn't disable the coil signal even when PD_VAR signals went larger than the threshold values PD_MAX_VAR.

Also we replaced the label by the correct name "C1SUSAUX" on a tag which was tied to the front end machine mounted on the new 1X5 rack.

[Valera / Kiwamu]

It was because of a loose connection. Pushing the connector solved the issue.

We really have to think about making reliable connections and strain reliefs.

(Joe, Yuta)

Background:
 We are currently working on getting rid of "white stuff" in MEDM screens.
 Today, we fixed OPLEV stuff, MCL filters, and time stamps.

What we did:
 1. Plugged in OPLEV cables to ADC2. (See this wiki page for wiring)

 2. Connected ADC2 and OPLEV in Simulink model and fixed MEDM screens for OPLEVs (Attached #1).

 3. Put MCL filters for BS,ITMX,ITMY,PRM,SRM.
  They don't need them, but just for getting rid of "white stuff."
  They are connected to the ground, so the outputs are always 0.

 4. Fixed "TIME_STRING"s in MEDM screens so that they show current time correctly.
  You only need to put text monitor with channel "C1:FEC-DCU_NODE_ID_TIME_STRING" into master files(DEFAULTNAME things) and run generate_master_screens.py.
  It will automatically sets DCU ID correctly!! (Great work, Joe!)
  See this wiki page for more info on making MEDM screens.

 5. Checked OPLEV for MC2 by pointing a laser pointer to QPD. (For MC2, OPLEV is just a transmission beam position monitor)
  Each quadrant looked like they are connected to the right channel numbers.

Plan:
 - figure out what C1:SUS-NAME_MODE_SW1 does and fix
 - fix Whitening, Dewhitening ON/OFF button in main MEDM screens, so that they switch every channels' filters
 - make a new screen for MC (like the old one C1IOO_ModeCleaner.adl)
 - create a new mark for new MEDM screens

Since MC2_TRANS is, in fact, MC2 Transmission, and not an oplev at all (it's not red, and it's not a lever, although it does use a QPD), I propose that the name be changed to something sensical, since calling it an oplev is completely non-sensical.  The name change should happen sooner rather than later, to avoid confusion.

Actually the clipping of POP wasn't in the chamber but it was on the first lens on the optical bench.

So I repositioned the lens to avoid the clipping and now there are no clipping on POP.

As with the previous post, I eliminated and unnecessary hop through c1rfm for the c1als --> c1mcs connection for the ALS output to MC2 POS.

As a side note, we might considering piping the ALS signals into the LSC input matrix, elevating them to actual LSC error signals, which in some since they are.  It's just that right now we're routing them directly to the actuators without going through the full LSC control.

(Joe, Yuta)

Summary:
 This Monday, MC suspension damping got something wrong.
 We started to check filters and found that digital filters were wrong because of mis-conversion from old filter files to new files.
 We converted the file again, and with mutual understanding between Foton and us, we finally got correct filters(I hope!).

What we did:
 1. Merged filter files in old /cvs/cds/caltech/chans/ directory into C1SUS.txt(BS,ITMX,ITMY), C1RMS.txt(PRM,SRM), C1MCS.txt(MC123).

 2. Rebuilt the RT models in order to get a correct filter file header(they have list of filter modules).

 3. Concatenate the header with filter design part which we got from step1.

 4. Replaced 'N 2048' with 'N 16384'
   It replaces sampling rate of "XXSEN"s.

Basically, step1-4 was the same with what we did last time. We didn't changed the fitler coefficients, so Foton somehow changed the original filter design.
So, this time, we

 5. Deleted coefficients like we did on Tuesday (see elog #3774).

But Foton couldn't read the file correctly this time. Foton seemed to be unbeatable.
Even if we replaced the sampling rate, Foton kept saying 2048! (This is maybe because Foton's default value is 2048Hz. Everytime Foton notice some editting in the file, he destroys everything. He hates editting)
The problems were always associated with the sampling rate, so

 6. Got back to step4 and undo-ed the replacement.

 7. Foton could read it this time, so I changed the sampling rate one by one using Foton GUI.

 8. Checked filters using Foton's Bode Plot.(Not for all, but some that had problem before)

 9. Splitted SDSEN filters to SDSEN, SUSSIDE, and SDCOIL.

 10. Put some missing whitening filters, and 28HzELPs.
   BS, PRM, SRM didn't have any 28HzELP for SDCOIL.
   ITMX and ITMY SDCOIL had SimDW/InvDW which doesn't make sense(SIDEs don't have analog DW). So, I deleted and replaced with 28HzELP.

F2A issue:
 We failed in sending F2A filters to new filter files.
 These are a little bit complicated because TO_COIL_X_X filters were named ULPOS,URPOS etc before.
 Also, MC3 didn't have any F2As, so maybe we should but the same F2As as MC1/2.
 Note that every F2As are different, and TO_COIL matrix have UL,UR,LL,LR order(not same as INMATRIX).
 Also, SRM had f2pv instead of F2A!

Next work:
 - Check whole filters by actually measuring transfer function between SENs and COILs.
 - Damp MC suspentions, and lock MC.
 - Measure openloop TF and compare with the designed.


How do you read a Foton filter file:
 When you open up a Foton filter file, you see filters like this.

################################################################################
### modulename                                                               ###
################################################################################
# SAMPLING modulename samplingrate
# DESIGN   modulename n filterdesign
# DESIGN   modulename n filterdesign
###                                                                          ###
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
modulename   n xy z      v      w filtername                        gain    a1     a2    b1     b2
                                                                            a1     a2    b1     b2
n: filter number
 0 for FM1, 1 for FM2, ... , 9 for FM10

x: Input Switching setting
 1 Always On
 2 Zero History

y: Output Switching setting
 1 Immediately
 2 Ramp
 3 Input Crossing
 4 Zero Crossing

z: number of filters cascaded.

v: if y=2, (Ramp Time(sec))*(samplingrate)
   if y=3 or 4, Tolerance

w: (Timeout(sec))*(samplingrate)

 Note that v and w are changed when sampling rate is changed.

 Transfer function will be;
  H(1/z)=G*(1+b1/z+b2/z/z)/(1+a1/z+a2/z/z)
  z=exp(s/fs)
 where fs is the sampling frequency.

Reference:
 Kiwamu Izumi: "Notes about Digital Filters," http://tamago.mtk.nao.ac.jp/izumi/green/DigitalFilter.pdf

(Rana, Yuta)

Summary:
 The damping servo for MCs has not been working since Monday.
 We already found that some filter coefficients were wrong.(see elog #3742)
 So, we fixed it (just for MCs now).
 But still doesn't work.

What we did:
Fixed the filters for MC suspensions;
 1. In /cvs/cds/rtcds/caltech/c1/chans/ directory, we replaced C1MCS.txt with ./filter_archive/c1mcs/C1MCS_101019_101927.txt.
  This is the archive of the filter bank for MC suspensions, when filter designs were correct(when we hadn't opened it with foton yet).

 2. Deleted all the filter coefficients.
   By using emacs magical C-<space> C-x r k.

 3. Loaded with foton to get correct coefficients.

 4. Reloaded coefficients for C1MCS.

We now have the correct filters, but the damping still doesn't work.

 5. To confirm that the filters are now correct and the model is working correctly, we measured the transfer function between ULSEN input and ULCOIL output and compared with the calculated TF from the filter bank file (Attachment #1) .

 6. To calculate the designed function, we used the following matlab scripts;
    /cvs/cds/caltech/users/rana/mat/utilities/FotonFilter.m        # takes the foton file and returns the TF
    /cvs/cds/caltech/users/rana/mat/utilities/readFilterFile.m    # reads the foton file
    /cvs/cds/caltech/users/yuta/scripts/sentocoil.m                   # calculates the total TF from SEN to COIL

Result:
 As you can see from the attachment, filters seem fine now. (The red line is the measured and the blue line is the calculated function in the plot)
 But the damping (for POS, PIT, YAW) still doesn't work. Why????

Next work:
 - see if coils are pushing correctly
 - push cable connectors further!
 - check input and output filters (whitening, dewhitening)
 - when fixed, use my fully updated QAdjuster.py for adjusting Q for multiple channels automatically!!

(Joe, Yuta)

Summary:
 This evening, we fixed output filter analog/digital switchings in Simulink logic, but they didn't actually switched filters.
 That was because what we thought BO0 was BO2 and BO2 was BO0.
 We re-wired, re-labeled the cables.
 We checked it by using a LED(see "test configuration" in this wiki page).
 Also, we checked the Simulink connection by probing MAX333A in SOS Dewhitening Board located at 1X4-8-11B, which switches SIDE coils for all optics.

Next work:
 - Currently, FM10 in UL/UR/LR/LL/SDCOIL filters switch analog/digital, but let's make FM9 do the switching.(See schematic in elog #3758)
 - Check all the logic by measuring transfer functions(maybe single frequency measurement is enough. we can use tdsdmd etc to automate).

(Joe, Yuta)

Summary:
 We found some mistakes in whitening filter switches yesterday(see elog #3748).
 One was the mis-connection between LL/LR and the second was the bit invert.
 So, we fixed it and checked that they are now switching correct by probing MAX333A.

Why we made mistake:
1. LL/LR mis-connection
  Simply, Simulink model was wrong.
  It occurred because UL/UR/LR/LL order is currently arbitrary and confusing.
  For now, we just swap the connection(Attached #1).
  In the future, we should fix all orders corresponding to the actual wiring order(UL > LL > UR > LR).

2. Bit invert
  It occurred from counter-intuitive output of Contec 32 BO(See Attachment #2(taken from this wiki page)).
  If "1," current goes from A to B, so MAX333A input will be 0V, which means analog WF is on.
  If "0," no current, so MAX333A input will be +15V, which means analog WF is off(bypassed).

Next work:
 - There are no digital WF in SDSEN inputs. So, we have to put them.
 - Dewhitening filters are totally messed up now. Switches are wrong, some optics have digital DW but some doesn't, some have 28Hz elliptic LPF.......
     We have to grand unify them.

Now the SRM Yaw bias in yaw is functional without any strage behavior.
The problem was found at the connector of the flat ribbon cable from the DAC to the cross connect.

I used the extender board to diagnose the SRM coil driver circuit at 1X4.
The UL coil input did not show any sign of voltage no matter how the bias slider was jiggled.

I opened the side panel of the rack and found the signal was absent at the cross connect which relays two flat ribbon cables
for the SRM coil driver. I checked the DAC output with a multimeter. All the bias outputs were OK at the DAC.

Then I opened the IDC connector at the DAC side of the crossconnect as the signal was already missing there.
I found that the flat ribbon cable was a half line shifted from the supposed location.
This resulted a short circuit of the DAQ +/- pins for the SRM UL coil.

I recrimped the connector and now the SRM Yaw slider is back.
This changed the nominal position of the SRM. The new slider values were saved.

I already have reported in this entry that REFL165 shows too high DC output which does not depend on the light level on the diode.
Today I removed REFL165 from the table and inspected it.

The diode has been burnt as shown in the first picture (left).
The window is smoked, and the photo sensitive surface has been removed from its base. It moves in the can.

The burnt diode was replaced to the new one.
The new one shows ~30% better capacitance of ~50pF and I had to increase the inductance from 14nH (i.e. 15nH//220nH) to 18nH.
After some struggles to increase/decrease the stray inductance by moving the SMD capacitors a little, the resonance is reasonably tuned to 166MHz.

The comprehensive test will be performed shortly.

 Yes.  I think that's a good idea, since we're not using them at this time, and we want c1mcs to behave.  Maybe make an elog note of which version is the first without them, so that we can conveniently find the model(s) in the svn?

Jenne just discovered an issue with the python-matplotlib package that I knew was coming but forgot about.

We pull packages from the LSCSOFT Debian "squeeze" archive, which is a convenient way for us to install LIGO data analysis software.  There are no LSCSOFT archives for Ubuntu, and Debian "squeeze" is the closest supported distribution to Ubuntu 10.04 "lucid", which is what we are using.

DASWG recently added python-matplotlib to the LSCSOFT squeeze archive.  The version they added (1.0.1-3) supersedes the version in lucid, so by default apt wants to install it.  However, the LSCSOFT version is compiled against a newer version of some standard libraries, so it won't function on our system and seg faults.

The solution (a solution) is to use apt "pinning" to pin the package to the lucid version that works.  I've added the following file on all the 10.04 workstations to prevent the package from upgrading to the LSCSOFT version:

controls@pianosa:~ 0$ cat /etc/apt/preferences.d/pin_python-matplotlib
Package: python-matplotlib
Pin: release a=lucid
Pin-Priority: 1000

controls@pianosa:~ 0$ cat /etc/apt/preferences.d/pin_python-matplotlib
Package: python-matplotlib
Pin: release a=lucid
Pin-Priority: 1000 

I forgot that there were a couple of different matplotlib packages that all needed to be pinned.  To be safe I decided to just pin all packages to the lucid versions.  This will still allow us to install lscsoft packages that are not ubuntu, but it will always prefer packages from lucid instead.  Here's the new pinning file:

controls@pianosa:~ 0$ cat /etc/apt/preferences.d/pinning 
Package: *
Pin: release a=lucid
Pin-Priority: 1000
controls@pianosa:~ 0$ 

I've started to try to fix all the old non-working medm snapshots.  I've made a new snapshot directory, and some new snapshot scripts to handle taking the snapshots, and view old ones.

Snapshots are now in:  /opt/rtcds/caltech/c1/medm/snap

There is one main script: /opt/rtcds/caltech/c1/medm/snap/snapcommands.  This is linked by:

• updatesnap: update a snapshot
• viewsnap: view most recent snapshot
• viewold: view old snapshots

These commands take either the path to the screen in question, or it's relative path to the medm directory (/opt/rtcds/caltech/c1/medm).  The snapshots for a specific screen are stored in a directory specific to the screen, in a place relative to the snap directory that mimics the screens relative path to the overall medm directory.  So for instance, the snap directory for: 

/opt/rtcds/caltech/c1/medm/c1lsc/master/C1LSC_OVERVIEW.adl

is:

controls@rossa:~ 0$ find /opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/2012-08-01-02:21:34-UTC.png
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/2012-08-01-02:21:04-UTC.png
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/2012-08-01-02:21:23-UTC.png
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/2012-08-01-03:41:34-UTC.png
/opt/rtcds/caltech/c1/medm/snap/c1lsc/master/C1LSC_OVERVIEW/current.png
controls@rossa:~ 0$ 

The "current.png" is a link to the most recent snapshot, and is what you get when you "viewsnap".

Below is an example medm snippet of what could be used in screens to enable this functionality.  I've fixed up a couple of the screens, but there are a lot more that need to be updated.

"shell command" {
    object {
        x=666
        y=597
        width=40
        height=40
    }
    command[0] {
        label="Settings and Procedures"
        name="emacs"
        args="./help/IFO_ALIGN.txt &"
    }
    command[1] {
        label="View Snapshot"
        name="/opt/rtcds/caltech/c1/medm/snap/viewsnap"
        args="MISC/IFO_ALIGN.adl &"
    }
    command[2] {
        label="View old snapshots"
        name="/opt/rtcds/caltech/c1/medm/snap/viewold"
        args="MISC/IFO_ALIGN.adl &"
    }
    command[3] {
        label="Update Snapshot"
        name="/opt/rtcds/caltech/c1/medm/snap/updatesnap"
        args="MISC/IFO_ALIGN.adl &"
    }
    clr=14
    bclr=30
}

Flow bench effect on oplev error signal     is here.

I turned off the south X-end flow bench.

The south end flow bench HEPA filter should be run all times. You can turn it off for a measurement or two but remember we are storing clean optics there.

The zero count bench will reach  room particle count  ~ 10,000 in one minute.

 My bad.  I turned it off last night to see if it would help make the Xgreen more stable, and then when I woke up this morning I realized that I had forgotten to turn it back on.  Bad Jenne.

I made a very useful flowchart for the week. Our goal for the week is to measure g-factor of PRC in PRMI.

Our fluorecent lights became obsolete.  We'll have change fixtures over to some more energy efficient one. Do you have any recommendation regarding to less noise performer unit?

We may go this direction of LED fluorecent lamps ?

I stole the foam EOM box and the temperature controller circuit from the PSL table so that I could continue with the RAM measurements here at the ATF.

That is all.

This box was made  to provide good thermal stability for seismometer calibration. There is an inner solid shell of 0.064" Al box that is covered by 2" insulation

inside and outside. The polystyrene foam is "CertiFoam 25 SE " from McMasterCarr #9255K3.

Aluminum foil replaced by sheet metal on Enclosure and AP table.

This morning (~0100) I started to redo some of the wiring in the rack with the FB in it. This was in an effort to activate the new Megatron (Sun Fire 4600) which we got from Rolf.

Its sitting right above the Frame Builder (FB). The fibers in there are a rats nest. Someone needs to team up with Joe to neaten up the cabling in that rack - its a mini-disaster.

While fooling around in there I most probably disturbed something, leading to the FB troubles today.

TP3 drypump replaced after 10,344 hrs at 750 mTorr foreline pressure.

The foreline pressure is 13 mTorr after 8 hrs of running, TP3: 50K rpm, 0.14 Amp with all annuloses pumped.

The annulos pressures are 0.3 - 5 mtorr

************************

Frame builder just crashed again

 It appears that foton does not like the unstable poles, which we need to model the transfer functions.

But one can try to load the filters into the front end by generating the filter file e.g.:

#
# MODULES DARM_ASDC 
#
################################################################################
### DARM_ASDC                                                                   ###
################################################################################
# SAMPLING DARM_ASDC  16384
# DESIGN   DARM_ASDC  
### ####
DARM_ASDC  0 21 6  0  0 darm 1014223594.005454063416 -1.95554205062071  0.94952075557861 0.06176931505784 -0.93823068494216
                         -2.05077577179611   1.05077843532639  -2.05854170261687  1.05854477394411
                         -1.85353637553024   0.86042048250739  -1.99996540107622  0.99996542454814 
                         -1.93464836371852   0.94008893626414  -1.89722830906561  0.90024221050918
                         -2.04422931770060   1.04652211283968  -2.01120153956052  1.01152717233685 
                         -1.99996545575365   0.99996548582538  -1.99996545573320  0.99996548582538

Unfortunately if you open and later save this file with foton it will strip the lhp poles.

 I've checked foton files for small numbers. There are several other filters besides "Cheby" that are presented with small numbers. For example, "BTW0.01" in the LOCKING_Q, LOCKING_I modules,  "SeisDaytime"  in the SUP_MC3_SP_NOISE and others. The output of the commands is presented below.

>chans

>cat C1???.txt | grep e-23

SUP_MC3_SP_Y_NOISE 2 12 4  16384      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC3_SP_YAW_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC3_SP_ROLL_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC3_SP_PIT_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC2_SP_Y_NOISE 2 12 4  16384      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC2_SP_YAW_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC2_SP_ROLL_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC2_SP_PIT_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC1_SP_Y_NOISE 2 12 4  16384      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC1_SP_YAW_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC1_SP_ROLL_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUP_MC1_SP_PIT_NOISE 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_MC3_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC3_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC3_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC3_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC3_LSC 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC2_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC2_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC2_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9978637592754149   0.9978663974923444   2.0000000000000000   1.0000000000000000

SUS_MC2_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC1_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC1_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_MC1_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_SRM_SUSYAW 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_SRM_SUSPOS 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_SRM_SUSPIT 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_PRM_SUSYAW 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_PRM_SUSPOS 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_PRM_SUSPIT 3 12 4  32768      0 Cheby         1.1175580413719e-23    -1.99726998010527     0.99727436063954     2.00000000000000     1.00000000000000

SUS_ETMX_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMX_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMX_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMX_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMX_LOCKIN1_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SUS_ETMX_LOCKIN1_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SUS_ETMX_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SUS_ETMX_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SUS_ETMY_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMY_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMY_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_ETMY_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SUS_PIT_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_ROLL_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_YAW_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_Y_NOISE_FILT 2 12 4  16384      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_PIT_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_ROLL_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_YAW_NOISE_FILT 2 21 4      0      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

SUS_Y_NOISE_FILT 2 12 4  16384      0 SeisDaytime    4.3736847644382e-23    -1.99994247406600     0.99994247737496    -0.00000000000000    -1.00000000000000

BS_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

BS_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

BS_LOCKIN1_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

BS_LOCKIN1_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

BS_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

BS_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

BS_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

BS_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMX_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMX_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMX_LOCKIN1_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMX_LOCKIN1_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMX_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMX_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMX_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMX_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMY_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMY_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMY_LOCKIN1_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMY_LOCKIN1_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

ITMY_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMY_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMY_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

ITMY_SUSYAW 3 21 4      0      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

PRM_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

PRM_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

PRM_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

PRM_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

PRM_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

PRM_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SRM_LOCKIN2_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SRM_LOCKIN2_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SRM_LOCKIN1_Q 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SRM_LOCKIN1_I 5 21 2      0      0 BTW0.01     1.351815922494362e-23  -1.9999929139431329   0.9999929139578397   2.0000000000000000   1.0000000000000000

SRM_SUSPIT 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SRM_SUSPOS 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SRM_SUSSIDE 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

SRM_SUSYAW 3 12 4  32768      0 Cheby       1.117558041371939e-23  -1.9972699801052749   0.9972743606395355   2.0000000000000000   1.0000000000000000

 Last night I found that the response of ITMX against the angle offsets were strage.

Eventually I found a loose connection at the feedthrough connectors of ITMX chamber.

So I pushed the connector hard, and then ITMX successfully became normal.

It looked like someone had accidentally kicked the cable during some works.

This bad connection had made unacceptable offsets in the OSEM readout, but now they seem fine.

This was the error today:

GET /40m/ HTTP/1.1
Host: nodus.ligo.caltech.edu:8080
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://www.ligo.caltech.edu/~ajw/40m_upgrade.html
Cookie: elmode=threaded; __utma=65601905.411937803.1291369887.1291369887.1291369887.1; __utmz=65601905.1291369887.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SITESERVER=ID=4981c5fd42ae53c9c9e0980f2072be4f

Found exactly the same error messages at the end of the log file.

(Koji, Jenne, Yuta)

We found one of DAC cables had a poor contact.
That probably caused our too much "tilt" of the beam into MC.

Story:
  From the previous A2L measurement and MC aligning, we found that the beam is somehow vertically "tilted" so much.
  We started to check the table leveling and the beam height and they looked reasonable.
  So, we proceeded to check coil balancings using optical levers.
  During the setup of optical levers, I noticed that VMon for MC1_ULCOIL was always showing -0.004 even if I put excitation to coils.
  It was because one of the DAC cables(labeled CAB_1Y4_88) had a poor contact.
  If I push it really hard, it is ok. But maybe we'd better replace the cable.

What caused a poor connection?:
  I don't know.
  A month ago, we checked that they are connected, but things change.

How to prevent it:
  I made a python script that automatically checks if 4 coils are connected or not using C1:IOO_LOCKIN oscillator.
  It is /cvs/cds/caltech/users/yuta/scripts/coilchecker.py.
  It turns off all 3 coils except for the one looking at, and see the difference between oscillation is on and off.
  The difference can be seen by demodulating SUSPOS signal by oscillating frequency.
  If I intentionally unplug CAB_1Y4_88, the result output for MC1 will be;

==RESULTS== (GPS:973512733)
MC1_ULCOIL      0.923853382664 [!]
MC1_URCOIL      38.9361304794
MC1_LRCOIL      55.4927575177
MC1_LLCOIL      45.3428533919

Plan:
 - Make sure the cable connection and do A2L and MC alignment again
 - Even if the cables are ok, it is better to do coil balancings. See the next elog.

The cause is apparent! The connectors on the cables are wrong!
Currently only 50% of the pin length goes into the connector!

[Koji / Yuta]

There were the guys who used the PENTEK 40pin connectors into the IDC 40pin connectors.
Those connectors are not compatible at all.

==> We replaced the connectors on the cables from DAC to IDC adapters to the dewhitening board for the vertex SUSs.

In addition, I found one of the Binary OUT IDC50pin connector has no clamp.

==> We put the IDC50pin clamp on it.

PENTEK connectors were inserted. The latches are not working!

the vertical pitch is different between PENTEK and IDC!

Wow! Where is the clamp???

The framebuilder is back online now, minus it's symmetricom GPS card.  The card seems to have failed entirely, and was not able to be made to work at downs either.  It has been entirely removed from fb.

As a fall back, the system has been made to work off of the system NTP-based time synchronization.  The latest symmetricom driver, which is part of the RCG 2.4 branch, will fall back to using local time if the GPS synchronization fails.  The new driver was compiled from our local checkout of the 2.4 source in the new to-be-used-in-the-future rtscore directory:

controls@fb ~ 0$ diff {/opt/rtcds/rtscore/branches/branch-2.4/src/drv/symmetricom,/lib/modules/2.6.34.1/kernel/drivers/symmetricom}/symmetricom.ko
controls@fb ~ 0$ 

The driver was reloaded.   daqd was also linked against the last running stable version and restarted:

controls@fb ~ 0$ ls -al $(readlink -f /opt/rtcds/caltech/c1/target/fb/daqd)
-rwxr-xr-x 1 controls controls 6592694 Dec 15 21:09 /opt/rtcds/caltech/c1/target/fb/daqd.20120104
controls@fb ~ 0$ 

We'll have to keep an eye on the system, to see that it continues to record data properly, and that the fb and the front-ends remain in sync.

The question now is what do we do moving forward.  CDS is not supporting the symmetricom cards anymore, and have moved to using Spectracom GPS/IRIG-B cards.  However, Downs has neither at the moment.  Even if we get a new Spectracom card, it might not work in this older Sun hardware, in which case we might need to consider upgrading the framebuilder to a new machine (one supported by CDS).

 You (Jamie) should talk with Rolf and Alex to find out what framebuilder they will support for > 3 years. Then we should buy that along with the adapter card which allows us to use the same RAID we now have for the frames.

The fsck on the framebuilder (fb) raid array (/dev/sda1) completed overnight without issue.  I rebooted the framebuilder and it came up without problem.

I'm now working on getting all of the front-end computers and models restarted and talking to the framebuilder now.

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

So the framebuilder/daqd frame writing issue seems to have been a transient one.  Alex said he tried enabling frame writing manually and it worked fine, so I tried re-enabling the frame writing config lines and sure enough it worked fine.  So it's a mystery.  Alex said the "main profiler warning" lines tend to show up when data is backed up in the buffer, although he didn't explain why exactly that would have been the issue here.

daqdrc frame writing directives:

start frame-saver;
sync frame-saver;
start trender;
start trend-frame-saver;
sync trend-frame-saver;
start minute-trend-frame-saver;
sync minute-trend-frame-saver;
start raw_minute_trend_saver;

Dataviewer is recovered after heavy new year party.

Alex and I have taken the framebuilder offline to try to see what's wrong with the symmetricom card.  We have removed the card from the chassis and Alex has taken it back to downs to do some more debugging.

We have been formulating some alternate methods to get timing to the fb in case we can't end up getting the card working.

There seems to be something strange going on with the 40m frame builder.
Specifically, there is a gap in the frames in /frames/full near the start of
each 100k second subdirectory. For example, frames for the following times are missing:

990200042-990200669
990300045-990300492
990400044-990400800
990500032-990500635
990600044-990600725
990700037-990700704
990800032-990800677
990900037-990900719

To summarize, after writing the first two frames in a data directory, the next ~10 minutes of frames are usually missing. To make matters worse (for
the nds2 frame finder, at least) the first frame after the gap (and all successive frames) start at an arbitrary time, usually not aligned to a 16-second boundary. Is there something about the change of directories that is causing the frame builder to crash? Or is the platform/cache disk too slow to complete the directory switch-over without loss of data?