40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 114 of 339  Not logged in ELOG logo
ID Date Author Type Category Subject
  11347   Mon Jun 8 15:51:31 2015 ericqUpdateCDSRCG Diagnostics

I've started making some model changes for RCG diagnostic tests. 

I put some blocks down in C1TST and C1RFM to test the delays of all-digital loops and one loop with a direct DAC->ADC (which currently uses a janky 1-pin lemo -> BNC -> 2-pin lemo situation (which will be improved)). 

Here's what C1TST looks like now. 

I've taken TFs of all three loops. The all digital loops are flat on the order of microdBs. 

The delay in loop A (single loop, one model) is consistent with one 16k cycle, plus or minus 0.25 nsec. 

The delay in loop C (single loop, two models connected via RFM) is consistent with two 16k cycles, plus or minus 0.5 nsec. 

I haven't yet grabbed the whitening and AA/AI shapes for loopB, to calibrate the real delay. 

All of these files currently live in /users/ericq/2015-06-CDSdiag, but I'll make somewhere outside of the user directory to collect all of these tests soon. 

Attachment 1: newTST.png
newTST.png
  11346   Fri Jun 5 11:59:59 2015 ericqBureaucracyGeneralMaintenance Tasks, IFO upgrades

At wednesday's meeting, Rana, Koji, Steve, and I started making a list of maintenence tasks that should be done/checked on a regular basis. The actual scheduling of these has not yet been considered. They include:

  • N2 Tank pressure / cylinder replacement
  • Headlamp, walkie talkie battery recharge
  • Workstation software updates
  • Coffee bean and filters
  • Multimeter battery levels
  • Sorensen DC power supply voltage settings and current draws
  • UPS' status (Vacuum, NFS host, workstations)
  • SR560s, battery powered scopes plugged in
  • Rack Fuses intact
  • Take pictures of electronics racks, optical tables
  • Replace PSL HEPA filter

Next, we brainstormed work that can be done to improve the interferometer performance, and what order/precedence they should take. 

In the end, it was decided that the plan for the next few weeks was to focus on improving the ALS noise levels, and, more importantly, seeking to make the performance more consistent. We need to know what is limiting us, and what we extent we can expect to improve things. To this end, I am working on reviving a ALS noise budget; using the noise budget from the green locking paper to inform a simulinkNB kind of thing.

Here are all of the items we listed during this brainstorming session.

Some near-term/priority tasks are:

  • Installing the accelerometers near MC1 and 2
  • Installing green steering PZT mirrors at the Y end table, commission dither alignment
  • Improving the X end green mode matching
  • ALS noise budgeting
  • Upgrade the realtime system to RCG v2.9

More down the line, other things we thought about were:

  • Cleanup of bench power supplies (FSS box has one, where else?)
  • Fixing the ETMX suspension issues 
  • Upgrade SOS suspension code to the appropriate aLIGO block
  • Upgrade to the green PDH electronics
  • Understanding / tuning the FSS servo laser PZT vs. PC crossover
  • Undestanding / tuning the 11MHz vs 55MHz modulation phase
  • Replacing the slow vxworks machines with the Acromag setup Aiden has set up for the CTN lab
  • QPD upgrade
  • New/better green beatbox
  • Finalize the manifestation of the IR beat control (Freq counter vs. fast DFD)
  • Explore the idea of using an analog output of the ALS beatbox as fast input to the CM board
  • Replace triple resonant EOM driving circuit with double resonant one
  • X end table layout/enclosure upgrade
  11345   Wed Jun 3 17:04:08 2015 ericqUpdatePEMWilcoxon Accelerometer Huddle Test

I've set up the Wilcoxon accelerometers on the SP table for a huddle test, to estimate their noise levels. 

They're clamped down to the table along the same axis, and their housings are in good contact, in hopes to make them as correlated as possible. 

Steve helped me move the DAQ cables from under the BS/PRM oplev table, to dropping from the cable tray above the POX table, so I could get them plugged in at the SP table. This also helps reduce the rats nest by the access connector, and is a fine location for when the accelerometers are attached at MC1 & MC2. 

A quick glance at DTT shows coherence of >0.9 from about 2-100Hz for all six. A three-corner-hat deal will provide an easy estimate of the noise floor of each one. If we feel like being fancy / accounting for possible gain differences, we could try some MISO wiener action, but that is probably overkill. 

Attachment 1: AccHuddle.jpg
AccHuddle.jpg
  11344   Wed Jun 3 11:55:52 2015 SteveUpdateOptical LeversPRM-BS oplev

I'm getting ready change the Newport Ultima U100-AC to SS-Polaris-K1  LOW DRIFT MIRROR MOUNTS

Note: there is only one lens in the PRM lunching path (  only realized later ) , so the spots are large ~ 3 mm at PRM qpd and ~4.5 mm at BS qpd

The spots are well centered.

Atm3,  the spots were well centered yesterday ( the PRM is misaligned in pitch and retsore does not work today )


 

Attachment 1: PRM-BS_oplEr_noLenses.png
PRM-BS_oplEr_noLenses.png
Attachment 2: PRM-BSopl_3jun2015.jpg
PRM-BSopl_3jun2015.jpg
Attachment 3: opServos.png
opServos.png
Attachment 4: 10d_trend_opl.png
10d_trend_opl.png
Attachment 5: opl_trend_40d.png
opl_trend_40d.png
  11343   Tue Jun 2 21:22:07 2015 rana, kojiConfigurationIOOAOM inserted in beam and aligned

We spent an hour today to put the AOM back in the beam before the PMC and verified that the diffraction is working.

  1. The fuse holder was missing from the rack. We inserted a 5A fuse. We expect that the quiesscent draw is < 0.5 A. The power is from the +24V Sorensen supply.
  2. The alignment was tricky, but we optimized it as well as we could in translation and the RZ direction. Its a fixed mount still.
  3. We noticed that according to the datasheet, the polarization is wrong! It wants S-Pol light and we're giving it P-Pol. How come no one noticed this? We expect that the efficiency is reduced because of this. We (Steve) need to brainstorm what kind of mount we can use there to mount it at 90 deg to the plane of the table.
  4. The lens after the AOM has f = +400 mm. The distance from the AOM to the lens is ~800-900 mm so its not so terrible. However, if someone were to put the AOM halfway between the turning mirrors there, the beam diffraction would be canceled.
  5. The AOM input impedance seems to be 50 Ohm as advertised. The previous Koji entry claim of 25 Ohm is mysterious. We checked the Ohmage by sending a signal into the AM input of the AOM using the DS345 which as a 50 Ohm output. 1 Vpp from the DS345 made 1 Vpp on the input of the AM input as measured by Oscope connected by T with high impedance setting.
  6. With 0.5 V offset and a 1 Vpp signal, we get ~20-25% modulation of the power.sad
  7. We have left it running with a 4444.4 Hz modulation and a small amplitude. This is to see if we can use this to measure the cavity poles of the MC and the arms.
  8. We noticed some hash on the Teed input monitor. It was backstreaming of the RF drive. Whoever uses this thing in an ISS feedback ought to make sure to put an RF choke between the servo and this AOM driver.

We also removed a 50/50 pickoff mirror which was used to take one of the NPRO -> EOM polarizer reject beams and send it across the table into a floppy dump. Its now hitting a closer floppy dump. Let's stop using these crappy anodized aluminum flappers anywhere, Steve.

We also noticed that the PMC REFL path uses a W1 from CVI to send the PMC reflection to the REFL RFPD. The dim beam from the AR coated surface is being used rather than the bright beam from the uncoated surface. Ooops. Steve, can you please order another W1 for 1064 from CVI, but get it with a 2-3 deg wedge angle? This one has a wedge which is too small.

  11342   Mon Jun 1 20:05:36 2015 ranaUpdateCDSRCG Upgrade Imminent

yes

Quote:

 

Additionally, we want to have some set of tests and diagnostics, to make sure we have not introduced unwanted behavior. 

To this end I will create some test models and DTT templates, where a series of measurements can be run, like

  • OLTF/delay measurement of a single all-digital loop within one model
  • OLTF/delay measurements of a few all-digital loops split across two models, using IPC communcation, RFM, dolphin
  • Hook up DAC -> resistor/amplifier/??? -> ADC, to check things like DAC output, ADC noise levels, IOP delays.

I'll run these test before touching anything, and make sure I understand all of the results, so that an apples-to-apples comparison can be made after the upgrade is complete. 

I got goosebumps just imagining this.

  11341   Mon Jun 1 15:22:19 2015 SteveUpdatePEMGuralps X- short cable is bad

 

Quote:

Koji and Steve,

The result: bad Guralp x-arm cable.

I will swap the short cables tomorrow at the base.

 

Short 46" long cables at the base plates were swapped. Their solderings looked horrible.

This cable actually worked at 5-5-2015

Bad cable at ETMY station now.  The new cable should be a little bit longer ~52"

Attachment 1: seismGur1-2.png
seismGur1-2.png
  11340   Mon Jun 1 10:26:53 2015 ericqUpdateCDSRCG Upgrade Imminent

We are planning on upgrading the 40 CDS system to the latest version of the LIGO RCG software, in three weeks when Jamie is back in town. 

Jamie and I spoke last week, to hash out a general plan for upgrade and preperations I can make in the meantime. 


Preparation of our models for the upgrade includes;

  • Check if any of the default RCG parts (filter modules, etc.) have substantially different default behavior / ports
  • Cleaning up unterminated/hanging connections in the diagrams (Jamie tells me RCG is more strict about this now)
  • Going through all of the build logs for our models to find what custom blocks are being pulled in from the userapps svn
  • Confirm all of our running blocks and models are committed to svn 
  • In a new, isolated, folder, checkout the latest version of the userapps repo
  • See what blocks have changed, and what model changes might be neccesary. 

Once we think we know what needs to be changed in our models, we can check out the latest version of the RCG source, without linking it as the active version, and create a new build directory, without touching the old one, and create new copies of the 40m models with the neccesary modifications. This way, we can work on getting all of the 40m models compiled, without touching any of the live, running, systems

Once our models are compiling successfully, we can work on building daqd, nds, mxstream, etc. 


Additionally, we want to have some set of tests and diagnostics, to make sure we have not introduced unwanted behavior. 

To this end I will create some test models and DTT templates, where a series of measurements can be run, like

  • OLTF/delay measurement of a single all-digital loop within one model
  • OLTF/delay measurements of a few all-digital loops split across two models, using IPC communcation, RFM, dolphin
  • Hook up DAC -> resistor/amplifier/??? -> ADC, to check things like DAC output, ADC noise levels, IOP delays.

I'll run these test before touching anything, and make sure I understand all of the results, so that an apples-to-apples comparison can be made after the upgrade is complete. 


Updates will be posted as I hash things out. I'm sure we have not yet thought of everything to think about and test, so ideas and feedback are very welcome. 

  11339   Mon Jun 1 08:32:14 2015 SteveUpdateSUSPRM damping restored

Local earthquake 3.8 Mag tripped only PRM

Vac monitor is not communicating.

PSL HEPA turned on
 

Attachment 1: indio3.8Meq.png
indio3.8Meq.png
  11338   Fri May 29 15:12:39 2015 KojiUpdateComputer Scripts / ProgramsChiara Backup Hiccup

Actual data

Attachment 1: backup_hours.pdf
backup_hours.pdf
  11337   Fri May 29 12:49:53 2015 KojiUpdateComputer Scripts / ProgramsChiara Backup Hiccup

In fact, the file access is supposed to be WAY faster now than in the RAID case.

As noted in ELOG 9511, it was SCSI-2(or 3?) that had ~6MB/s thruput. Previously the backup took ~2hours.
This was improved to 30min by SATA HDD on llinux1.

I am looking at /opt/rtcds/caltech/c1/scripts/backup/rsync.backup.cumlog

In fact, this "30-min backup" was true until the end of March. After that the backup is taking 1h~1.5h.

This could be related to the recent NFS issue?

  11336   Fri May 29 11:28:42 2015 ericqUpdateComputer Scripts / ProgramsChiara Backup Hiccup

I've changed the chiara local backup script to read a folder exclusion file, and excluded /users/public_html/detcharsummary, and things are working again. 

This was neccesary because the summary pages are being updated every half hour, which is faster than the time it takes for the backup script to run, so the file index that it builds at the start becomes invalid later on in the process. 


Thinking about chiara's disk, it strikes me that when we went from the linux1 RAID to a single HDD on chiara, we may have tightened a bottleneck on our NFS latency, i.e. we are limited to that single hard drive's IO rates. This of course isn't the culprit for the more recent dramatic slowdowns, but in addition to fixing whatever has happened more recently, we may want to consider some kind of setup with higher IO capability for the NFS filesystem. 

  11335   Fri May 29 02:05:08 2015 ericqUpdateLSCEnd Laser temperatures set

Both green beatnotes have been found with nominal amplitudes. (X: -30dBm Y: -20dBm), at temperatures which don't seem to be prone to mode hopping. 

X = 42.64, Y ~40.15

Both arms can lock on ALS, but as Koji mentioned in ELOG 11334, the ALS noise seems anomalously high. 


Details

The temperatures I posted in the previous log ended up not being so useful. To find the right end laser temperatures, I looked at the IR beatnotes on the control room analyzer out to 1GHz, and swept around the SLOW_SERVO2_OFFSET channels to change the laser temperature. During this time, the end green shutters were closed, the PSL shutter was closed, the FSS slow servo was off, and the FSS_SLOWDC was set to 0.0. (The green PSL shutter had to be open, because the IR beat fiber is coupled after it.)

For each arm, I found three temperatures where an IR beat could be observed; as Koji mentioned on Wednesday, we should use the middle mode. For each of the arms, scanning around from the middle to move the beat by +-1GHz did not cause a mode hop - the beat stayed visible on the scope. Once I found a real IR beat for the X arm, I took at look at the RF output of the X BBPD, and found my alignment from the other night was actually pretty good; I made a minor touch up to maximize the green beat. 

For the AUX X innolight, I was able to find the actual temperature of the laser crystal when at the correct point, remove the digital offset, and return to the same temperature with the controller front panel dial. This temperature is 42.64 degrees Celcius. There is no diode temperature control on the unit, as far as I could tell, but the maximum green transmitted power through the X arm is about the same as it's ever been, around GTRX=0.5. 

My motivation for doing this was that I always have problems remembering the historical typical values for the digital temperature offset. It seems much cleaner to me to set things up such that a beat is visible "at the origin" (i.e. FSS_SLOWDC=0 and SLOW_SERVO2_OFFSET=0) I suppose that this will also depend on what mode of the PMC we're locked to. During my time working today, its been locked near the middle of its range, currently sitting at 145V. 

However, for the AUX Y lightwave, I was a little perplexed to find that moving the the laser crystal setpoint around does not apear to change the real laser temperature at all. Thus I could not offload the digital offset in the same way. Aditionally, the lightwave controller does not have the same temperature measuring accuracy as the innolight, even with the back panel voltage readout that is hooked up to the multimeter that lives under the optical table. The best I can tell is around 40.1-40.2 degrees C. Y_SLOW_SERVO2_OFFSET of -10690 counts gives a beat <100MHz at FSS_SLOWDC=0. This is actually very close to the previous operating point, so the same mode seems to still work. 

The arms now easily lock on ALS, albeit with higher noise. With arms locked on ALS, POX and POY show >1kHz rms noise. frown 


I gave the PRFPMI locking script a few tries, but it's having problems keeping the PRMI locked. The ALS is a few times noisier than usual, and I haven't revisited the validity of the PRC angular feedforward, so I'm not so surprised. 

  11334   Thu May 28 21:10:46 2015 KojiUpdateGreen LockingALS-X noise hunting

I have been looking at the X-end ALS setup.
I was playing with the control bandwidth to see the effect to the phase tracker output (i.e. ALS err).
For this test the arm was locked with the IR and the green beat note was used as the monitor.

From the shape of the error signal, the UGF of the green PDH was ~10kHz. When I increased the gain
to make the servo peaky, actually the floor level of the ALS err became WORSE. I did not see any improvement
anywhere. So, high residual error RMS cause some broadband noise in the ALS??? This should be checked.
Then when the UGF was lowered to 3kHz, I could see some bump at 3kHz showed up in the ALS error.
I didn't see the change of the PSD below 1kHz. So, more supression of the green PDH does not help
to improve the ALS error?

Then, I started to play with the phase tracker. It seems that someone already added the LF booster
to the phase tracker servo. I checked the phase tracker error  and confirmed it is well supressed.
Further integrator does not help to reduce the phase tracker error.

For the next thing I started to change the offset of the phase tracker. This actually changes
the ALS error level!
The attached plot shows the dependence of the ALS error PSD on the phase tracker
output. At the time of this measurement, the offset of -10 exhibited the best noise level.
This was, indeed, factor of 3~5 improvement compared to the zero offset case below 100Hz.

I'm afraid that this offset changes the beat frequency as I had the best noise level at the offset of -5
with a different lock streatch. We should look at this more carefully. If the beat freq changes the offset,
this give us another reason to fix the beat frequency (i.e. we need the frequency control loop.

= Today's ALSX error would have not been the usual low noise state.
We should recover the nominal state of the ALS and make the same test =

Attachment 1: 150528_ALS.pdf
150528_ALS.pdf
  11333   Thu May 28 17:12:32 2015 ericqUpdateIOOFSS SLOW not engaged: is this intentional?

Yes, I had turned it off while looking for the PSL/X AUX beat, and forgot to turn it back on.

I will post an elog with more detail this evening, but I found a temperature which restored the X green beatnote at its nominal amplitude (-30dBm) with no mode hops within +-1 IR beat GHz, and offloaded the slow offset slider to the X-end laser crystal dial. I will look for the Y beatnote after dinner. 

Currently the control room analyzer is hooked up to recieve the Y IR and green beats; no X signals. 

  11332   Thu May 28 17:00:04 2015 KojiUpdateIOOFSS SLOW not engaged: is this intentional?

I found that FSS SLOW servo is not engaged. Is this intentional test to keep the NPRO temp constant?
This is making the FSS Fast unhappy (~ -7.5V right now).

  11331   Thu May 28 16:43:52 2015 SteveUpdatePEMGuralps swapped

Koji and Steve,

The result: bad Guralp x-arm cable.

I will swap the short cables tomorrow at the base.

 

Attachment 1: GursSwapped.png
GursSwapped.png
  11330   Thu May 28 15:10:44 2015 ranaUpdatePEMSeismic Confusion

You are plotting the STS channels, not Guralp. These are for the Trillium 240 seismometer.

Also, you cannot tell if the seismometer is working by plotting the MEAN trend. That just gives the average and we need the fluctuations. Better off looking at the spectrum like I did last time.

And....its not good enough to just do the bubble. You have to do the mass centering procedure that you and I did last time with the breakout paddle.

  11329   Thu May 28 10:42:19 2015 SteveUpdatePEMbad Guralp X-cable
Quote:

Tried swapping cables at the Guralp interface box side. It seems that all of our seismic signal problems have to do with the GUR2 cable being flaky (not surprising since it looks like it was patched with Orange Electrical tape!! rather than proper mechanical strain relief).

After swapping the cables today, the GUR2 DAQ channels all look fine: i.e. GUR1 (the one at the Y end) is fine, as is its cable and the GUR2 analog channels inside the interface box.

OTOH, the GUR1 DAQ channels (which have GUR2 (EX) connected into it) are too small by a factor of ~1000. Seems like that end of the cable will need to be remade. Luckily Jenne is still around this week and can point us to the pinout / instructions. Looks like there could be some shorting inside the backshell, so I've left it disconnected rather than risk damaging the seismometer. We should get a GUR1 style backshell to remake this cable. It might also be possible that the end at the seismometer is bad - Steve was supposed to swap the screws on the granite-aluminum plate on Thursday; I'll double check.

The Guralps were swapped.

What I did:   turned DC power off at 1X1,  hand carried them,  centered  each leveling bubbles, gently locked the jack screws and turned power back on.

ETMY at the east end now has CMG-T40-0008, sn T4157 as channel C1:PEM-SEIS_STS_1_Y_OUT_DQ.........

ETMX at south end now has CMG-T40-0053, sn T4Q17 as channel C1:PEM-SEIS_STS_2_Y_OUT_DQ.........

Conclusion:  Guralps are fine.  X  cable is bad. It was bad 6 months ago when it was made.

We can still swap the 3ft short cables at the granite base if Rana has not done it.

 

Attachment 1: Gurs180dXbad.png
Gurs180dXbad.png
Attachment 2: swappedGurs.png
swappedGurs.png
  11328   Wed May 27 17:14:08 2015 ericqUpdateLSCX Aux Laser crystal temperature changed

Rana suspects that the lack of X beatnote is related to the PSL laser temperature change (ELOG 11294). 

I used the information on the wiki and old elogs (wiki-40mELOG 6732), to deduce that the new end laser temperatures should be:

  • X end-> 38.98 C
  • Y end-> 35.80 C

I went out to the X end and found the laser crystal temperature set to 40.87, which is not what the measurements I linked to suggest would be the ideal temperature for the previous NPRO laser temperature of 30.89, which would be 37.02. I could not find any elog describing the choice of this setpoint.  

I've changed the X end laser crystal temperature to the value above. I've hooked up the X IR and green beatnotes to go the control room analzyer, and have been looking for the beatnote as I adjust the digital temperature offset, but haven't found it yet...

If this proves totally fruitless, we can just put the lasers back to their original temperatures, since it's unclear if it helped the PC drive noise levels. 

  11327   Wed May 27 15:20:54 2015 ericqUpdateComputer Scripts / ProgramsChiara Backup Hiccup

The local chiara backups are still failing due to vanished source files. I've emailed Max about the summary page jobs, since I think they're running remotely. 

  11326   Wed May 27 02:53:57 2015 ericqUpdateGeneralifo recovery log

Given my suspicion of fault with the X Green BBPD, Koji generously provided me with another one that he had confirmed to be working. 

However, I turns out I was mistaken. With the existing BBPD, I did indeed witness a beat in the RF output, but it is somehow something like 20dBm smaller than it should be. This is why I missed it the other night. Here's a video of a RF output on a scope, wherein the beat is only barely visible because I've set the trigger level very low. I could not make the beatnote any larger through any alignment motions; I had gotten to this point by doing near/far field overlap on the PSL table. 

I'm not sure what could have caused this. Mode mismatch? By eye, the beam spots looked about the same in the near and far fields, and we haven't had to touch the mode matching in quite some time... I've given up on trying to solve this for tonight. 


Just for kicks, I hooked up the fiber PD IR beatnotes as inputs to the ALS DFD. The X beat is too small to even really see above the control room analyzer's noise floor, but the Y beat looked big enough. With the arms locked on IR, the phase tracker output RMS was a few kHz, so not even worth thinking about any further. Not so surprising. 

Finally, I put back / hooked up everything in its nominal state. 

  11325   Tue May 26 19:57:11 2015 ranaUpdateComputer Scripts / ProgramsifoCoupling

Looks like a very handy code, especially with the real statistical tests.

I would make sure to use much smaller excitation amplitudes. Since the coupling is nonlinear, we expect that its only a good noise budget estimator when the excitation amplitude is less than a factor of 3 above the quiesscent excitation.

  11324   Tue May 26 11:05:10 2015 SteveUpdatePEMworked around ETMY seism.

The cable tray holder cross beam is removed and cut short for good access to seismometer.

Attachment 1: ETMYseismic.png
ETMYseismic.png
Attachment 2: cut.jpg
cut.jpg
  11323   Sun May 24 14:45:02 2015 KojiHowToGeneralHow to launch StripTools at specified locations

LLO Operator Tips:

koji.arai@cr9:/opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment$ cat autostart_strips.sh 
 

#!/bin/bash

# Baffle window setup 1500x480

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/baffle_pd.stp &
sleep 1

# DC signals setup

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+470' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/dc_signals.stp &
sleep 1

# WFS prx mich sry setup

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0-24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/wfs_prx_mich_sry.stp &
sleep 1


exit

  11322   Sat May 23 22:43:10 2015 ericqUpdateGeneralifo recovery log

Running train-of-thought elog:


East N2 cylinder found empty, replaced. West is >2kpsi


Removed Yuta-specific code from damprestore. A grep for 'yuta' in the python files within /scripts/ shows some other occurances, but nothing that is really in use at this time. New feature of damprestore.py: remembers oplev status.


Ran LSCoffsets. 

WFS offsets relieved (all <20).

Adjusted FSS offset to minimize MC_FAST_MON

ASS ran (but the arm alignment has been astoundingly stable lately. I haven't touched it all this week)

ITMX is the only optic that got a correction over 20 counts. 

BS and *TM oplev spots look well centered, except for ITMX. 

I undid the gain reduction rana introduced because X ASS seemed to be really slow. It is currently fine in its older state. What's going on here?


Some network latency stuff is going on, even freezing up terminals when trying to write text files. This may (or may not) be correlated with the summary page rysnc jobs on nodus. It occurs to me that we have a DAQ ethernet network seperate from the martian network, but the frame transfers need to go through the martian network, since nodus is the only way out to the outside world. If we had a machine/gateway directly from the DAQ network to the caltech network, the martian network wouldn't get bogged down when frames are being uploaded


GTRY = 0.55, ok. Aligned GTRX to 0.52, also ok

Y beatnote was found easily. Have spent >30 minutes looking for X green beatnote. Typical FSS slow and X temperature ranges don't seem to be giving much. Will check the beat alignment with a scope, but if the beat is too high to begin with, it may not work...


I suspect a problem with the X Green BBPD

I could see the IR beatnote between the PSL and AUX X lasers at the input to the frequency counter. (I believe it is a real beatnote because it reacted as expected to the end temperature moving, and stabilizing the end laser to the arm). However, when placing the IR beatnote at a frequency which should've made the green beatnote visible on an analyzer and/or scope, no beatnote was found. I played with the beat alignment to no avail; the DC output of the BBPD behaved as expected, but I never saw anything in the RF output or on the control room analyzer. sad I also checked the beatnote signal chain by hooking up a 1mV 26MHz signal into the cable that hooks up to the RF out of the BBPD; the signal showed up clearly on the control room analyzer. 

I'm not sure what may have happened. ELOG 9996 may be related. 

I'm calling it a night. 

  11321   Fri May 22 18:09:58 2015 ericqUpdateComputer Scripts / ProgramsifoCoupling

I've started working on a general routine to measure noise couplings in our interferometers. Often this is done with swept sine measurements, but this misses the nonlinear part of the coupling, especially if the linear part is alreay reduced through some compensation or feedforward scheme. Rana suggested using a series of narrow band-limited noise injections. 

The structure I'm working on is a python script that uses the AWG interface written by Chris W. to create the excitations. Afterwards, I calculate a series of PSD estimates from the data (i.e. a spectrogram), and apply a two-sample, unequal variance, t-test to test for statisically significant increases in the noise spectra to try and evaluate the nonlinear contriubutions to the noise. I've started a git repository at github.com/e-q/ifoCoupling with the code. 

So far, I've tested one such injection of noise coupling from the ETMX oplev error point to the single arm length error signal. It's completely missing the user interface and structure to do a general series of measurements, but this is just organizational; I'm trying to get the math/science down first. 

Here's a result from today:

Median, instead of the usual mean, PSDs are used throughout, to reject outliers/glitches.

The linear part of the coupling can be estimated using the coherence / spectrum height in the excitation band, but I'm not sure what the best what to present/paramerize the nonlinear parts of each individaul excitation band's result is.

Also, I anticipate being able to write an excitation auto-leveling routine, gradually increasing the exctiation level until the excited spectrum is some amount  noisier than the baseline spectrum, up to some maximum amount configurable by the user. 

The excitation shaping could probably be improved, too. It's currently and elliptic + butterworth bandpass for a sharp edge and rolloff. 

I'm open to any thoughts and/or suggestions anyone may have!

Attachment 1: ETMX_PIT_L_coupling.png
ETMX_PIT_L_coupling.png
  11320   Fri May 22 12:09:57 2015 ranaUpdateSUSDampRestore script problem

I will move it back. We need to fix our scripts to not use any users/ libraries ever again.

Quote:

PRM watchdog tripped, but the damprestore.py script wouldn't run. 

It turns out the script tries to import some ezca stuff from /users/yuta (angry), which had been moved to /users/OLD/yuta (crying). 

I've moved the yuta directory back to /users/ until I fix the damprestore script. 

 

  11319   Fri May 22 11:59:54 2015 ericqUpdateSUSDampRestore script problem

PRM watchdog tripped, but the damprestore.py script wouldn't run. 

It turns out the script tries to import some ezca stuff from /users/yuta (angry), which had been moved to /users/OLD/yuta (crying). 

I've moved the yuta directory back to /users/ until I fix the damprestore script. 

  11318   Wed May 20 11:41:59 2015 ericqUpdateGeneralsome status

West cyclinder is empty, east is at 2000 psi; regulated N2 pressure is 64psi. I'll replace the west one after the meeting.

  11317   Wed May 20 03:08:27 2015 ranaUpdateGeneralsome status

I think that the real clue was that the dropouts are in some channels and not in others:

https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150519/psl/

As it turns out, the channel with no dropouts is the RAW PSL RMTEMP channel. All the others are the minute trends. So something is up with the trend making or the trend reading in the cluster.

Quote:

There's a few hours so far after today's c1cal shut off that the summary page shows no dropouts. I'm not yet sure that this is related, but it seems like a clue.

 

  11316   Tue May 19 19:24:30 2015 ranaUpdatePEMSeismic BLRMS filters

I was wondering about the design of the BLRMS fitlers for the seismic channels since the STS ones seem to have so little gain compared to the Guralps.

Here are some plots of the Bode magnitude and impulse responses of the bandpass filters (before the low passing). There's a bunch of entries from Masha on this from her SURF summer. Can anyone comment on why they are all so different?

One of the old Masha entries speaks of designing the lowpass filter in an intelligent way: by adjusting the filter order until the power in the stopband is less than 1% of the power in the passband. Seems like we could do that for bandpass too. For now I have made the names reasonable and changed all of the BP filters to 4th order Butterworth.

Also, it turns out that the Vel2Vel (gain ~0.02) filters were mistakenly on in the STS BP filter banks. The GUR inputs have a gain to scale the counts to velocity, but the STS seem to already be in microns/sec (where is this gain?) so I turned off and deleted the Vel2Vel filters; in any case the gain should not be done seperately in each BP bank, but altogether before the BP filtering.

Attachment 1: BLRMS_BP.pdf
BLRMS_BP.pdf
Attachment 2: BLRMS_imp.pdf
BLRMS_imp.pdf
  11315   Tue May 19 18:55:12 2015 rana, ericQUpdateGeneralsome status

After one day the pressures are east/west = 2200/450 PSI

Quote:

 

  1. New east pressure reading is 2500 PSI. Regulated N2 pressure is 68 PSI.

 

  11314   Tue May 19 18:38:33 2015 ranaUpdateCDSMXstream restart script working (beta)

Good catch; that was some seriously bad programming on my part. I had some undeclared variable garbage going on. I fixed it and re-implemented the script in CRON on megatron. The log file shows that it has detected no problems for the last several checks. I'll check it again tomorrow to see if its doing good or bad.

  11313   Tue May 19 17:53:38 2015 ranaUpdateModern ControlBrushing up on Wiener Filtering

Good to see that misofw.m is still alive and well. Todo:

  1. Apply weighting to the filter via time domain SOS prefiltering of the signals (see Jenne's code from the LLO Global FF paper)
  2. Consider using some of the MC SUSPOS signals. Its not strictly legal, but I wonder if its responsible for out noise below 1 Hz.
  3. Steve was in the midst of hooking up the Wilcoxones to get better subtraction at 5-15 Hz, but couldn't find cables. We need to hunt them down in W Bridge.
  4. Attach one of the medium or big Lings to the MC2 chamber platform and see if we can do this in hardware without driving the suspension. WE want to use a DAC channel (from where?) and pipe it to the Ling using a medium high current drive. Might start with the 50 Ohm output of the SR560 and later use a BUF634 ala Crackle coil driver.
  11312   Tue May 19 17:03:34 2015 KojiUpdateCDSMXstream restart script working (beta)

AutoMX is resetting mx_stream every 5 minutes. Basically everytime AutoMX is called,
it resets mx_stream. Is the mx_stream stalling such often? Or the script is detecting false alerms?


> tail -200 /opt/rtcds/caltech/c1/scripts/cds/autoMX.log

Tue May 19 16:43:01 PDT 2015
LSC - FB bad. Runnning restart:
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1sus closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1lsc closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1ioo closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscex closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscey closed.
0
Tue May 19 16:48:02 PDT 2015
LSC - FB bad. Runnning restart:
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1sus closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1lsc closed.
ssh_exchange_identification: read: Connection reset by peer
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscex closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscey closed.
0
Tue May 19 16:53:01 PDT 2015
LSC - FB bad. Runnning restart:
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1sus closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1lsc closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1ioo closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscex closed.
 * Stopping mx_stream ...                                                 [ ok ]
 * Starting mx_stream ...                                                 [ ok ]
Connection to c1iscey closed.

  11311   Tue May 19 16:18:57 2015 ericqUpdateGeneralcrons fixed

I wrapped rampdown.py in rampdown.sh, which is just these lines:

#!/bin/bash
source /ligo/cdscfg/workstationrc.sh
/opt/rtcds/caltech/c1/scripts/SUS/rampdown.py > /dev/null 2>&1

This is now what megatron's cron runs. It appears to be working.

I also added the workstationrc line to the n2 and chiara HDD checking scripts that run on nodus, which should resolve the issue from ELOG 11249

  11310   Tue May 19 14:51:44 2015 ericqUpdateModern ControlBrushing up on Wiener Filtering

As part of preparing for the SURF projects this summer, I grabbed ~50 minutes of MCL and STS_1 data from early this morning to do a little MISO wiener filtering. It was pretty straightforward to use the misofw.m code to achieve an offline subtraction factor of ~10 from 1-3Hz. This isn't the best ever, but doesn't compare so unfavorably to older work, especially given that I did no prefiltering, and didn't use all that long of a data stretch.

Code and plot (but not data) is attached. 

Attachment 1: mclData.png
mclData.png
Attachment 2: mclWiener.zip
  11309   Tue May 19 11:50:52 2015 manasaUpdatePEMNo noticeable effect from M4.0 earthquake

There was an earthquake: M4.0 - 40km SSW of South Dos Palos, California 

No noticeable effects on the IFO. MC did not lose lock; however the arms did unlock.

  11308   Tue May 19 11:24:44 2015 ericqUpdateComputer Scripts / ProgramsNotification Scheme

Given some of the things we've facing lately, it occurs to me that we could be better served by having some sort of unified human-alerting scheme in place, for things like:

  • Local/offsite backup failures
  • Vaccumm system problems
  • HDD status for things like /frames/ and /cvs/cds/, whether the disks are full, or their SMART status indicates imminent mechanical failure

Currently, many of these things are just checked sporadically when it occurs to someone to do so, or when debugging random issues. Smoother IFO operation and peace of mind could be gained if we're confident that the relevant people are notified in a timely manner. 

Thoughts? Suggestions on other things to monitor, like maybe frontend/model crashes?

  11307   Tue May 19 11:15:09 2015 ericqUpdateComputer Scripts / ProgramsChiara Backup Hiccup

Starting on the 14th (five days ago) the local chiara rsync backup of /cvs/cds to an external HDD has been failing:

caltech/c1/scripts/backup/rsync_chiara.backup.log:

2015-05-13 07:00:01,614 INFO       Updating backup image of /cvs/cds
2015-05-13 07:49:46,266 INFO       Backup rsync job ran successfully, transferred 6504 files.
2015-05-14 07:00:01,826 INFO       Updating backup image of /cvs/cds
2015-05-14 07:50:18,709 ERROR      Backup rysnc job failed with exit code 24!
2015-05-15 07:00:01,385 INFO       Updating backup image of /cvs/cds
2015-05-15 08:09:18,527 ERROR      Backup rysnc job failed with exit code 24!
...
 

Code 24 apparently means "Partial transfer due to vanished source files."

Manually running the backup command on chiara worked fine, returning a code of 0 (success), so we are backed up. For completeness, the command is controls@chiara: sudo rsync -av --delete --stats /home/cds/ /media/40mBackup

Are the summary page jobs moving files around at this time of day? If so, one of the two should be rescheduled to not conflict. 

  11306   Tue May 19 00:19:23 2015 ranaUpdateGeneralsome status

There's a few hours so far after today's c1cal shut off that the summary page shows no dropouts. I'm not yet sure that this is related, but it seems like a clue.

Attachment 1: Screen_Shot_2015-05-19_at_12.17.39_AM.png
Screen_Shot_2015-05-19_at_12.17.39_AM.png
  11305   Mon May 18 18:03:12 2015 ranaUpdateGeneralsome status

The c1cal model was maxing out its CPU meter so I logged onto c1lsc and did 'rtcds c1cal stop'. Let's see if this changes any of our FB / DAQD problems.

Attachment 1: CPUtrend.png
CPUtrend.png
  11304   Mon May 18 17:44:30 2015 ranaHowToCDSBypassing the CDSUTILS prefix issue

Too weird. I undid me changes. We'll have to make the C1: stuff work inside each python script.

Quote:
Quote:

export IFO=''

This makes things act weird:

controls@pianosa|MC 1> z avg 1 "C1:LSC-TRY_OUT"
IFO environment variable not specified.

 

  11303   Mon May 18 17:42:14 2015 rana, ericQUpdateGeneralsome status

Today at 5 PM we replaced the east N2 cylinder. The east pressure was 500 and the west cylinder pressure was 1000. Since Steve's elogs say that the consumption can be as high as 800 per day we wanted to be safe.

  1. We closed the black valve before the regulator and closed the valve on the cylinder.
  2. We unscrewed the brass fill line to the cylinder.
  3. We unchained the cylinder and put in the dolly (and attached the chains on there).
  4. We rolled in a fresh cylinder from outside using the red dolly (it should have chains).
  5. We put it in place, hooked up the chains, and screwed on the brass nozzle with the large adjustable wrench (need to put a non-adjustable here).
  6. Opened up the cylinder valve.
  7. Opened up the black valve.
  8. New east pressure reading is 2500 PSI. Regulated N2 pressure is 68 PSI.
Quote:

1) Checked the N2 pressures: the unregulated cylinder pressures are both around 1500 PSI. How long until they get to 1000?

 

  11302   Mon May 18 16:56:12 2015 ericqHowToCDSBypassing the CDSUTILS prefix issue
Quote:

export IFO=''

This makes things act weird:

controls@pianosa|MC 1> z avg 1 "C1:LSC-TRY_OUT"
IFO environment variable not specified.

  11301   Mon May 18 16:28:18 2015 ericqUpdateGeneralsome status
Quote:

4) Noticed that DAQD is restarting once per hour on the hour. Why?

It looks like daqd isn't being restarted, but in fact crashing every hour.

Going into the logs in target/fb/logs/old, it looks like at 10 seconds past the hour, every hour, daqd starts spitting out:

[Mon May 18 12:00:10 2015] main profiler warning: 1 empty blocks in the buffer                                     
[Mon May 18 12:00:11 2015] main profiler warning: 0 empty blocks in the buffer                                     
[Mon May 18 12:00:12 2015] main profiler warning: 0 empty blocks in the buffer                                     
[Mon May 18 12:00:13 2015] main profiler warning: 0 empty blocks in the buffer
...
***CRASH***

An ELOG search on this kind of phrase will get you a lot of talk about FB transfer problems. 

I noticed the framebuilder had 100% usage on its internal, non-RAID, non /frames/, HDD, which hosts the root filesystem (OS files, home directory, diskless boot files, etc), largely due to a ~110GB directory of frames from our first RF lock that had been copied over to the home directory. The HDD only has 135GB capacity. I thought that maybe this was somehow a bottleneck for files moving around, but after deleting the huge directory, daqd still died at 4PM. 

The offsite LDAS rsync happens at ten minutes past the hour, so is unlikely to be the culprit. I don't have any other clues at this point. 

  11300   Mon May 18 14:46:20 2015 manasaSummaryGeneralDelay line frequency discriminator for FOL error signal

Measuring the voltage noise and frequency response of the Analog Delay-line Frequency Discriminator (DFD)

The schematic and an actual photo of the setup is shown below. The setup was checked to be physically sturdy with no loose connections or moving parts.

The voltage noise at the output of the DFD was measured using an SR785 signal analyzer while simultaneously monitoring the signal on an oscilloscope.

The noise at the output of the DFD was measured for no RF input and at several RF input frequencies including the zero crossing frequency and the optimum operating frequency of the DFD (20MHz).

The plot below show the voltage noise for different RF inputs to the DFD. It can be seen that the noise level is slightly lower at the zero crossing frequency where the amplitude noise is eliminated by the DFD.

I also did measurements to obtain the frequency response of the setup as the cable length difference has changed from the prior setup. The cable length difference is 21cm and the obtained linear signal at the output of the DFD extends over ~ 380MHz which is good enough for our purposes in FOL. A cosine fit to the data was done as before. //edit- Manasa: The gain of SR560 was set to 20 to obtain the data shown below//

Fit Coefficients (with 95% confidence bounds):
       a =     -0.8763  (-1.076, -0.6763)
       b =       3.771  (3.441, 4.102)

Data and matlab scripts are zipped and attached.

Attachment 4: DFD.zip
  11299   Mon May 18 14:22:05 2015 ericqUpdateComputer Scripts / Programsrsync frames to LDAS cluster
Quote:

Still seems to be running without causing FB issues.

I'm not so sure. I just was experiencing some severe network latency / EPICS channel freezes that was alleviated by killing the rsync job on nodus. It started a few minutes after ten past the hour, when the rysnc job started. 

Unrelated to this, for some odd reason, there is some weirdness going on with ssh'ing to martian machines from the control room computers. I.e. on pianosa, ssh nodus fails with a failure to resolve hostaname message, but ssh nodus.martian succeeds. 

  11298   Mon May 18 11:59:07 2015 ranaUpdateGeneralsome status

Yes - my rampdown.py script correctly ramps down the watchdog thresholds. This replaces the old rampdown.pl Perl script that Rob and Dave Barker wrote.

Unfortunately, cron doesn't correctly inherit the bashrc environment variables so its having trouble running.

On a positive note, I've resurrected the MEDM Screenshot taking cron job, so now this webpage is alive (mostly) and you can check screens from remote:

https://nodus.ligo.caltech.edu:30889/medm/screenshot.html

ELOG V3.1.3-