* Relocked IMC. I guess it was stuck somewhere in the autlocker loop. I disabled autolocker and locked it manually. Autolocker has been reenabled and seems to be running just fine.

* The X arm has been having trouble staying locked. There seemed to be some amount of gain peaking. I reduced the gain from 0.007 to 0.006.

*  I disabled the triggered BounceRG filter : FM8 in the Xarm filter module.  We already have a triggered Bounce filter: FM6 that takes care of the noise at bounce/roll frequencies. FM8 was just adding too much gain at 16.5Hz. Once this filter was disabled the X arm lock has been much more stable. 
Also, the Y arm doesn't use FM8 for locking either.

1) Checked the N2 pressures: the unregulated cylinder pressures are both around 1500 PSI. How long until they get to 1000?

2) The IMC has been flaky for a day or so; don't know why. I moved the gains in the autolocker so now the input gain slider to the MC board is 10 dB higher and the output slider is 10 dB lower. This is updated in the mcdown and mcup scripts and both committed to SVN. The trend shows that the MC was wandering away after ~15 minutes of lock, so I suspected the WFS offsets. I ran the offsets script (after flipping the z servo signs and adding 'C1:' prefix). So far powers are good and stable.

3) pianosa was unresponsive and I couldn't ssh to it. I powered it off and then it came back.

4) Noticed that DAQD is restarting once per hour on the hour. Why?

5) Many (but not all) EPICS readbacks are whiting out every several minutes. I remote booted c1susaux since it was one of the victims, but it didn't change any behavior.

6) The ETMX and ITMX have very different bounce mode response: should add to our Vent Todo List. Double checked that the bounce/roll bandstop is on and at the right frequency for the bounce mode. Increased the stopband from 40 to 50 dB to see if that helps.

7) op340 is still running ! The only reason to keep it alive is its crontab:

op340m:SUS>crontab -l

07 * * * * /opt/rtcds/caltech/c1/burt/autoburt/burt.cron >> /opt/rtcds/caltech/c1/burt/burtcron.log
#46 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo > /cvs/cds/caltech/logs/scripts/FSSslow.cronlog 2>&1
#14,44 * * * * /cvs/cds/caltech/conlog/bin/check_conlogger_and_restart_if_dead
15,45 * * * * /opt/rtcds/caltech/c1/scripts/SUS/rampdown.pl > /dev/null 2>&1
#10 * * * *  /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1
#27 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/RCthermalPID.pl >/cvs/cds/caltech/logs/scripts/RCthermalPID.cronlog 2>&1

00 0 * * * /var/scripts/ntp.sh > /dev/null 2>&1
#00 4 * * * /opt/rtcds/caltech/c1/scripts/RGA/RGAlogger.cron >> /cvs/cds/caltech/users/rward/RGA/RGAcron.out 2>&1
#00 6 * * * /cvs/cds/scripts/backupScripts.pl
00 7 * * * /opt/rtcds/caltech/c1/scripts/AutoUpdate/update_conlog.cron
00 8 * * * /opt/rtcds/caltech/c1/scripts/crontab/backupCrontab

added a new script (scripts/SUS/rampdown.py) which decrements every 30 minutes if needed. Added this to the megatron crontab and commented out the op340m crontab line. IF this works for awhile we can retire our last Solaris machine.

8) To see if we could get rid of the wandering PCDRIVE noise, I looked into the NPRO temperatures: was - T_crystal = 30.89 C, T_diode1 = 21 C, T_diode2 = 22 C. I moved up the crystal temp to 33.0 C, to see if it could make the noise more stable. Then I used the trimpots on the front of the controller to maximimize the laseroutput at these temperatures; it was basically maximized already. Lets see if there's any qualitative difference after a week. I'm attaching the pinout for the DSUB25 diagnostics connector on the back of the box. Aidan is going to help us record this stuff with AcroMag tech so that we can see if there's any correlation with PCDRIVE. The shifts in FSS_SLOW coincident with PCDRIVE noise corresponds to ~100 MHz, so it seems like it could be NPRO related.

For some reason, my email address is the one that megatron complains to when cron commands fail; since 11:15PM last night, I've been getting emails that the rampdown.py line is failing, with the super-helpful message: expr: syntax error

Yes - my rampdown.py script correctly ramps down the watchdog thresholds. This replaces the old rampdown.pl Perl script that Rob and Dave Barker wrote.

Unfortunately, cron doesn't correctly inherit the bashrc environment variables so its having trouble running.

On a positive note, I've resurrected the MEDM Screenshot taking cron job, so now this webpage is alive (mostly) and you can check screens from remote:

https://nodus.ligo.caltech.edu:30889/medm/screenshot.html

Measuring the voltage noise and frequency response of the Analog Delay-line Frequency Discriminator (DFD)

The schematic and an actual photo of the setup is shown below. The setup was checked to be physically sturdy with no loose connections or moving parts.

The voltage noise at the output of the DFD was measured using an SR785 signal analyzer while simultaneously monitoring the signal on an oscilloscope.

The noise at the output of the DFD was measured for no RF input and at several RF input frequencies including the zero crossing frequency and the optimum operating frequency of the DFD (20MHz).

The plot below show the voltage noise for different RF inputs to the DFD. It can be seen that the noise level is slightly lower at the zero crossing frequency where the amplitude noise is eliminated by the DFD.

I also did measurements to obtain the frequency response of the setup as the cable length difference has changed from the prior setup. The cable length difference is 21cm and the obtained linear signal at the output of the DFD extends over ~ 380MHz which is good enough for our purposes in FOL. A cosine fit to the data was done as before. //edit- Manasa: The gain of SR560 was set to 20 to obtain the data shown below//

Fit Coefficients (with 95% confidence bounds):
       a =     -0.8763  (-1.076, -0.6763)
       b =       3.771  (3.441, 4.102)

Data and matlab scripts are zipped and attached.

It looks like daqd isn't being restarted, but in fact crashing every hour.

Going into the logs in target/fb/logs/old, it looks like at 10 seconds past the hour, every hour, daqd starts spitting out:

An ELOG search on this kind of phrase will get you a lot of talk about FB transfer problems. 

I noticed the framebuilder had 100% usage on its internal, non-RAID, non /frames/, HDD, which hosts the root filesystem (OS files, home directory, diskless boot files, etc), largely due to a ~110GB directory of frames from our first RF lock that had been copied over to the home directory. The HDD only has 135GB capacity. I thought that maybe this was somehow a bottleneck for files moving around, but after deleting the huge directory, daqd still died at 4PM. 

The offsite LDAS rsync happens at ten minutes past the hour, so is unlikely to be the culprit. I don't have any other clues at this point. 

Today at 5 PM we replaced the east N2 cylinder. The east pressure was 500 and the west cylinder pressure was 1000. Since Steve's elogs say that the consumption can be as high as 800 per day we wanted to be safe.

1. We closed the black valve before the regulator and closed the valve on the cylinder.
2. We unscrewed the brass fill line to the cylinder.
3. We unchained the cylinder and put in the dolly (and attached the chains on there).
4. We rolled in a fresh cylinder from outside using the red dolly (it should have chains).
5. We put it in place, hooked up the chains, and screwed on the brass nozzle with the large adjustable wrench (need to put a non-adjustable here).
6. Opened up the cylinder valve.
7. Opened up the black valve.
8. New east pressure reading is 2500 PSI. Regulated N2 pressure is 68 PSI.

The c1cal model was maxing out its CPU meter so I logged onto c1lsc and did 'rtcds c1cal stop'. Let's see if this changes any of our FB / DAQD problems.

There's a few hours so far after today's c1cal shut off that the summary page shows no dropouts. I'm not yet sure that this is related, but it seems like a clue.

I wrapped rampdown.py in rampdown.sh, which is just these lines:

#!/bin/bash
source /ligo/cdscfg/workstationrc.sh
/opt/rtcds/caltech/c1/scripts/SUS/rampdown.py > /dev/null 2>&1

This is now what megatron's cron runs. It appears to be working.

I also added the workstationrc line to the n2 and chiara HDD checking scripts that run on nodus, which should resolve the issue from ELOG 11249

After one day the pressures are east/west = 2200/450 PSI

I think that the real clue was that the dropouts are in some channels and not in others:

https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150519/psl/

As it turns out, the channel with no dropouts is the RAW PSL RMTEMP channel. All the others are the minute trends. So something is up with the trend making or the trend reading in the cluster.

West cyclinder is empty, east is at 2000 psi; regulated N2 pressure is 64psi. I'll replace the west one after the meeting.

Running train-of-thought elog:

East N2 cylinder found empty, replaced. West is >2kpsi

Removed Yuta-specific code from damprestore. A grep for 'yuta' in the python files within /scripts/ shows some other occurances, but nothing that is really in use at this time. New feature of damprestore.py: remembers oplev status.

Ran LSCoffsets. 

WFS offsets relieved (all <20).

Adjusted FSS offset to minimize MC_FAST_MON

ASS ran (but the arm alignment has been astoundingly stable lately. I haven't touched it all this week)

ITMX is the only optic that got a correction over 20 counts. 

BS and *TM oplev spots look well centered, except for ITMX. 

I undid the gain reduction rana introduced because X ASS seemed to be really slow. It is currently fine in its older state. What's going on here?

Some network latency stuff is going on, even freezing up terminals when trying to write text files. This may (or may not) be correlated with the summary page rysnc jobs on nodus. It occurs to me that we have a DAQ ethernet network seperate from the martian network, but the frame transfers need to go through the martian network, since nodus is the only way out to the outside world. If we had a machine/gateway directly from the DAQ network to the caltech network, the martian network wouldn't get bogged down when frames are being uploaded

GTRY = 0.55, ok. Aligned GTRX to 0.52, also ok

Y beatnote was found easily. Have spent >30 minutes looking for X green beatnote. Typical FSS slow and X temperature ranges don't seem to be giving much. Will check the beat alignment with a scope, but if the beat is too high to begin with, it may not work...

I suspect a problem with the X Green BBPD

I could see the IR beatnote between the PSL and AUX X lasers at the input to the frequency counter. (I believe it is a real beatnote because it reacted as expected to the end temperature moving, and stabilizing the end laser to the arm). However, when placing the IR beatnote at a frequency which should've made the green beatnote visible on an analyzer and/or scope, no beatnote was found. I played with the beat alignment to no avail; the DC output of the BBPD behaved as expected, but I never saw anything in the RF output or on the control room analyzer.  I also checked the beatnote signal chain by hooking up a 1mV 26MHz signal into the cable that hooks up to the RF out of the BBPD; the signal showed up clearly on the control room analyzer. 

I'm not sure what may have happened. ELOG 9996 may be related. 

I'm calling it a night. 

LLO Operator Tips:

koji.arai@cr9:/opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment$ cat autostart_strips.sh 
 

#!/bin/bash

# Baffle window setup 1500x480

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/baffle_pd.stp &
sleep 1

# DC signals setup

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0+470' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/dc_signals.stp &
sleep 1

# WFS prx mich sry setup

StripTool -xrm 'StripTool.StripGraph.geometry:1500x470+0-24' /opt/rtcds/userapps/trunk/asc/l1/scripts/initial_alignment/wfs_prx_mich_sry.stp &
sleep 1

exit

Given my suspicion of fault with the X Green BBPD, Koji generously provided me with another one that he had confirmed to be working. 

However, I turns out I was mistaken. With the existing BBPD, I did indeed witness a beat in the RF output, but it is somehow something like 20dBm smaller than it should be. This is why I missed it the other night. Here's a video of a RF output on a scope, wherein the beat is only barely visible because I've set the trigger level very low. I could not make the beatnote any larger through any alignment motions; I had gotten to this point by doing near/far field overlap on the PSL table. 

I'm not sure what could have caused this. Mode mismatch? By eye, the beam spots looked about the same in the near and far fields, and we haven't had to touch the mode matching in quite some time... I've given up on trying to solve this for tonight. 

Just for kicks, I hooked up the fiber PD IR beatnotes as inputs to the ALS DFD. The X beat is too small to even really see above the control room analyzer's noise floor, but the Y beat looked big enough. With the arms locked on IR, the phase tracker output RMS was a few kHz, so not even worth thinking about any further. Not so surprising. 

Finally, I put back / hooked up everything in its nominal state. 

At wednesday's meeting, Rana, Koji, Steve, and I started making a list of maintenence tasks that should be done/checked on a regular basis. The actual scheduling of these has not yet been considered. They include:

• N2 Tank pressure / cylinder replacement
• Headlamp, walkie talkie battery recharge
• Workstation software updates
• Coffee bean and filters
• Multimeter battery levels
• Sorensen DC power supply voltage settings and current draws
• UPS' status (Vacuum, NFS host, workstations)
• SR560s, battery powered scopes plugged in
• Rack Fuses intact
• Take pictures of electronics racks, optical tables
• Replace PSL HEPA filter

Next, we brainstormed work that can be done to improve the interferometer performance, and what order/precedence they should take. 

In the end, it was decided that the plan for the next few weeks was to focus on improving the ALS noise levels, and, more importantly, seeking to make the performance more consistent. We need to know what is limiting us, and what we extent we can expect to improve things. To this end, I am working on reviving a ALS noise budget; using the noise budget from the green locking paper to inform a simulinkNB kind of thing.

Here are all of the items we listed during this brainstorming session.

Some near-term/priority tasks are:

• Installing the accelerometers near MC1 and 2
• Installing green steering PZT mirrors at the Y end table, commission dither alignment
• Improving the X end green mode matching
• ALS noise budgeting
• Upgrade the realtime system to RCG v2.9

More down the line, other things we thought about were:

• Cleanup of bench power supplies (FSS box has one, where else?)
• Fixing the ETMX suspension issues 
• Upgrade SOS suspension code to the appropriate aLIGO block
• Upgrade to the green PDH electronics
• Understanding / tuning the FSS servo laser PZT vs. PC crossover
• Undestanding / tuning the 11MHz vs 55MHz modulation phase
• Replacing the slow vxworks machines with the Acromag setup Aiden has set up for the CTN lab
• QPD upgrade
• New/better green beatbox
• Finalize the manifestation of the IR beat control (Freq counter vs. fast DFD)
• Explore the idea of using an analog output of the ALS beatbox as fast input to the CM board
• Replace triple resonant EOM driving circuit with double resonant one
• X end table layout/enclosure upgrade

The summary pages have been down due to incompatibilities with a software update and problems with the LDAS cluster. I'm working at the moment to fix the former and the LDAS admins are looking into the latter. Overall, we can expect the pages will be fully functional again by Monday.

The pages are live again. Please allow some time for the system to catch up and process missed days. If there are any further issues, please let me know.
URL reminder: https://nodus.ligo.caltech.edu:30889/detcharsummary/

It was brought to my attention that the "Code status" page (https://nodus.ligo.caltech.edu:30889/detcharsummary/status.html) had been stuck showing "Unknown status" for a while.
This was due to a sync error with LDAS and has now been fixed. Let me know if the issue returns.

I made several small, nit-picky changes to the summary pages.

Motivation:

I'm still working on getting used to editing the summary pages. I also wanted to change some of the easy-to-alter cosmetics of the pages.

What I did:

I changed axis ranges, axis labels, and typos throughout the summary pages. Read below for an excrutiating list of the minor details of my alterations, if you wish:

• Changed axes on LSC control signals plots on the Summary tab (but will probably change these back to their original state)
• Moved an OpLev plot from the Sandbox tab to "Eve" tab
• Increased the y axis range on IOO MC2 Trans QPD and IMC REFLY RFPD DC plots (which may change when I better incorporate triggers into these plots)
• Fixed title on IOO Whitened Spectrogram and Rayleigh Spectrogram
• Fixed degree sign on Weather: Temperature and PSL Table Temperature
• Fixed percent sign on Weather: Humidity
•  

Results:

So far, everything looks good. I'll continue to make more changes later this week and hope to soon get on to more substatial changes.

I closed the PSL/GREEN shutters and shut off the LSC feedback/SUS watch dogs at 9AM PDT, to allow Jamie to start his disruptive work.
 

Today, I installed the Wilcoxon accelerometers in the table near the end of the mode cleaner. I only set three of them up instead of all six. They were set up just as Rana suggeted we should have them properly set up, i.e. cables being tighten up, and a box on top to prevent any airflow introducing any disturbances. We are planning on running the huddle test on these guys once the upgrade? to the interferometer is done.

The cables were tightly clamped to the table as shown below, I used a thick piece of shock absorbing rubber to do this.

 A small piece of thin rubber was used to hold each of the accelerometers tightly to the table in order not to damage them.

We had to borrow Megan's and Kate's piece of black foam in order to seal one of the sides properly, as the cable had to come out through somewhere. We didn't want to mess with drilling any holes into the box! 

There was a small crack even after using the foam. I sealed it up with duck tape.

The box isn't perfect, so there were multiple cracks along the bottom and top of it that could potentially allow for air to flow to the inside. Eric suggested that we should be super careful this time and do it right, so every crack was sealed up with ducktape.

Finally, we needed something heavy to be placed on top of the box to hold everything well. We used Rana's baby to accomplish this goal.

Just kidding! Rana's baby is too delicate for our purposes. A layman box of heavy cables was used instead. 

I re-measured the transfer function for Cable B, because the residuals in my previous post for cable B indicated a bad fit. 

I also realized I had made a mistake in calculating the time delay, and calculated more reasonable time delays today. 

Cable A had a delay of 202.43 +- 0.01 ns.

Cable B had a delay of 202.44 +- 0.01 ns. 

Here is the summary of my investigation how to set up my own "summary page" environment on the LDG (LIGO Data Grid) cluster.

Here all albert.einstein must be replaced with your own LIGO.ORG user name.

1. Obtain LDAS cluster account

Run the following from any of the terminal and use your LIGO.ORG credential
ssh albert.einstein@ssh.ligo.org
You will be suggested to visit a particular web site. Fill the form on the web site and wait for the approval e-mails.

2. Use LDG SSH login portal

Once you received the approval of the account, you should be able to log in to the system. Type the following command again from your local terminal
ssh albert.einstein@ssh.ligo.org
You are asked to select the site and machines. Select 2- CIT and b. ldas-pcdev1, c. ldas-pcdev2, or d. ldas-pcdev3.

3. Setup bash environment

Setting up the python library path is very important for the proper processing.

Here is my setup for .bash_profile

and .bashrc

The order of cleanpath and PYTHONPATH= is important as we want to use the local library installation before anything kicks in.

4. Install required Python libraries

Run the following lines with this order so that they are installed in your "~/local"

5. Setup summary pages for the 40m

Copy summary page setting from Max's directory.

cp -r ~max.isi/summary ~/

And make temporary directory for the summary pages.

mkdir /usr1/albert.einstein/summary

6. Modify typos in gw_summary_custon

Use your own editor to fix typos

emacs ~/summary/bin/gw_daily_summary_custom

replace max.isi to albert.einstein
change summary40m -> summary

Now the installation is done. From here, the description is for the routine procedure.

7. Run your summary page code

Run Kerberos authentication
kinit albert.einstein@LIGO.ORG

Run a summary page code for a specific date (e.g. for Jul 1st, 2015)
bash ${HOME}/summary/bin/gw_daily_summary_custom --day 20150701

The result can be checked under

https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/
https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/20150701/

Rerun a code for a specific page. This requires the page structure already exists.
The red texts should be modified depending on what ini file you want to run for what day.

/home/albert.einstein/.local/bin/gw_summary day --on-segdb-error warn --verbose  --output-dir . --multi-process 20 --no-html  --ifo C1 --archive C1EVE 20150630  --config-file /mnt/qfs2/albert.einstein/public_html/summary/etc/defaults.ini,/mnt/qfs2/albert.einstein/public_html/summary/etc/c1eve.ini

This command can actually be found in
https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/gw_summary_pipe.sh

8. Some useful command

To check which python library is used
python -c 'import gwpy; print gwpy.__file__'

To list installed python libraries and versions
pip list

This should return the list like the following.

...
astropy (1.0.3)
...
gwpy (0.1b1.dev121)
gwsumm (0.0.0.dev854)
...
matplotlib (1.4.3)
...
numpy (1.9.2)
...
scipy (0.15.1)
...

The summary pages are currently unstable due to priority issues on the cluster*. The plots had been empty ever since the CDS updated started anyway. This issue will (presubmably) disappear once the jobs are moved to the new 40m shared LDAS account by the end of next week.

*namely, the jobs are put on hold (rather, status "idle") because we have low priority in the processing queue, making the usual 30min latency impossible.

I'd like to build off of Koji's instructions with a few useful tips I discovered while setting up my own summary page environment.

To only make a specified selection of tabs for the summary pages, copy only the corresponding .ini files into /home/albert.einstein/summary/config and run the gw_daily_summary_custom following Koji's instructions below. When asked for nodus's password either hit "enter" three times without providing the password or comment out this section of the code to stop the summary page creation process from taking current data files from nodus. This is especially helpful when the 40m is down (like it is now).

After running the summary page code, the pages can be viewed at https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/YYYYMMDD/ and corresponding error logs can be found at ~/public_html/summary/logs/gw_summary_pipe_local-687496-0.err.

Over the past few days, I've been thinking about how to workout the details conerning Rana's request about a 'map' of the vicinity of the 40m interferometer. This map will take the positions of N randomly placed seismic sensors as well as the signals measured by each one of them and the calculated cross correlations between the sensors and between the sensors and the test mass of interest to give out a displacement vector with new sensor positions that are close to optimum for better seismic (and Newtonian) noise cancellation.

Now, I believe that much of the mathematical details have been already work out by Jenne in her thesis. She explains that the quantity of interest that we wish to minimize in order to find an optimal array is the following,

where   is the cross-correlation vector between the seismic detectors and the seismic (or Newtonian) noise,  is the cross-correlation matrix between the sensors and  is the seismic (or Newtonian) noise variance. 

I looked at the paper that Jenne cited from which she obtained the above quantity and noted that it is a bit different as it contains an extra term inside the square root, it is given by

where the new term,  is the matrix describing the self noise of the sensors. I think Jenne set this term to zero since we can always perform a huddle test on our detectors and know the self noise, thus effectively subtracting it from the signals of interest that we use to calculate the other cross correlation quantities.

Anyways, the quantity  above is a function of the positions of the sensors. In order to apply it to our situation, I'm planning on:

     1) Performing the huddle tests on our sensors, redoing it for the accelerometers and then the seismometers (once the data aquisition system is working... )  

     2) Randomly (well not randomly, there are some assumptions we can make as to what might work best in terms of sensor placement) place the sensors around the interferometer. I'm planning on using all six Wilcoxon 731A accelerometers, the two Guralps and the STS seismometer (any more?).

     3) Measure the ground signals and use wiener filtering in order to cancel out their self noises.

     4) From the measured signals and their present positions we should be able to figure out where to move the sensors in order to optimize subtraction.

i have also been messing around with Jenne's code on seismic field simulations with the hopes of simulating a version of the seismic field around the 40m in order to understand the NN of the site a little better... maybe. While the data aquisition gets back to a working state, I'm planning on using my simulated NN curve as a way to play around with sensor optimization before its done experimentally.

i have as well been thinking and learning a little bit about source characterization through machine learning methods, specially using neural networks as Masha did back in her SURF project on 2012. I have also been looking at Support vector machines. The reasons why I have been looking at machine learning algorithms is because of the nature of the everchanging seismic field around the interferometer. Suppose we find a pretty good sensor array that we like. How do we make sure that this array is any good at some time t after it has been found? If the array mostly deals with the usual seismic background (quiet) of the site of interest, we could incorporate machine learning techniques in order to mitigate any of the more random disturbances that happen around the sites, like delivery trucks, earthquakes, etc.

I applied a bandpass filter to the accelerometer huddle data as a pre-filter. The passband was from 5 Hz to 20 Hz. I found that applying this pre-filter did very little when comparing the PSD after pre-filtering to the PSD with no pre-filtering. There was some improvement though, just not a significant amount. For some reason, it also seemed as though the second accelerometer improved the most from pre-filtering the data, while the first and third remained closer to the unfiltered noise. Also, I have not yet figured out a consistent method for choosing passband ripple and stopband attentuation, both of which determine how good the filter is. 

My next step in pre-filtering will be determining a good method for choosing passband ripple and stopband attenuation, along with implementing other pre-filtering methods to combine with the bandpass filter. 

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 

As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

We need to update the indicators on the CDS_FE_STATUS screen to expose the new indicators, so that we have better visibility for these issues.

I'm not sure why this DAC is failing. It may indicate an actual problem with the DAC itself.

This is related to changes to how the front ends load their safe.snaps. I think they're now explictly expecting the file:

targtet/<model>/<model>epics/burt/safe.snap

I'll come over this afternoon and we can get acquainted with the new SDF system that now handles management of the safe.snap files.

I updated the bandpass filter I was using, finding that having different stopband attenuations before and after the passband better emphasized the area from 3 Hz to 20 Hz. I chose a low passband ripple but high stopband attenuation to do this. My passband ripple was 2 dB, the first stopband was 25 dB, and the second stopband attenuation was 40 dB. As can be seen in the filter Magnitude plot, this resulted in a fairly smooth passband and a fairly step dropoff to the stopband, which will better emphasize the region I am trying to isolate. My goal was to emphasize the 3-20 Hz region 10-30 times more than the outside regions. I think I accomplished this by looking at the Bode plot, but I may have chosen the second stopband attenuation to be slightly too high for this. 

Jamie showed me how to use the SDF system. We created new safe.snap files for all of the running models based on the autoburts from the morning of July 1st, before the upgrade began, and then pruned them of invalid channels. 

Now all of the models start up without having to race for the BURT button. 

We saw that c1sus was timing out all over the place once the filter settings had been restored. I was thinking I would move one of the vertex optics into c1mcs, but instead I found it easier to remove the global damping parts. Now the c1sus model runs at ~50usec.

The c1sus frontend's DAC is still nonfunctional. Jamie is seeking advice. 

On Thursday, new huddle test data for the Wilcoxon 731A was aquired by Eric. 

The difference between this new data and the previous data, is:

1) We used three accelerometers instead of six this time around.

2) We used a foam box, and clamped cables on the experimental set up as shown in the previous elog, http://nodus.ligo.caltech.edu:8080/40m/11389

I have analyzed the new data. Here I present my results.

The following plot shows the ASD's for the three accelerometers raw outputs as well as their error signals computed using the three cornered hat method,

As before, I computed the mean for the output signals of the accelerometers above as well as their mean self noise to get the following plot

,

Now, below I compare the new results with the results that I got from the old data, 

Did the enclosure and cable clamping do much? Not really, according to the computed three hat results. Also, notice how much better, even if its a small improvement, we get from using six accelerometers and calculating their self noise by the six cornered hat method.

Now, I moved on to analyzing the same data with Wiener Filtering.

Here are again, the raw outputs, and the self noises of each individual accelerometer calculated using Wiener filtering,

The accelerometer in the Y direction is show a kind of funky signal at low frequncies. Why? Anyways, I calculated the mean of the above signals as I did for the three cornered hat method above to get the following, I also show the means of the signals computed with the old data using wiener filtering,

Is the enclosure really doing much? The Wiener filter that I applied to the huddle test old data gave me a much better, by an order of magnitude better self noise curve. Keep in mind that this was using SIX accelerometers, not THREE as we did this time. I want to REDO the huddle test for the WIlcoxon accelerometers using SIX accelerometers with the improved experimental setup to see what I get.

Finally, I compare the computed self noises above with what the manufacturer gives,

,

As I expected, the self noise using six accelerometers and Wiener filtering is the best I could work out. The three cornered hat method works out pretty well from 1 to 10 Hz, but the noise is just too much anywhere higher than 10 Hz. The enclosed, clamped, 3 accelerometer wiener filter result is an order of magnitude worse than the six accelerometer wiener filtered result, and two orders of magnitude worse than the three cornered hat method in the 1 to 10 Hz frequency band. 

As I stated, I think we must performed the huddle test with SIX accelerometers and see what kind of results we get.

We took data for the mode cleaner a while ago, June 30th I believe. This data contained signals from the six accelerometers and the three seismometers. In here I have only focused on the seimometer signals as witnesses in order to construct Wiener filters for each of the three seismometer signals (x,y,z) and for the combined seismometers signal. The following plot showing the ASD's shows the results,

 Wiener filtering works beautifully for the seismometers. Note that subtraction is best when we use all three seismometers as the witnesses in the Wiener filter calculation, as can be clearly seen in the first plot above.

Now, I used vectfit to conver the Wiener FIR filters for each seismometer to their IIR versions. The following are the bode plots for the IIR filters,

For the x-direction seismometer,

For the y-direction seismometer

,

And for the z-direction seismometer,

 The IIR filters were computed using 5 zeros and 5 poles using vectfit. That was the maximum number of poles that I could use wihtout running into trouble with matrices being almost singular in Matlab. I still need to figure out how to deal with this issue in more detail as fitting the y-seismometer was a bit problematic. I think having a greater number of poles will make the fitting a bit easier.

(updateAfter Eric gave me feedback on my previous elog post, I went back and fixed some of the silly stuff I stated.

First of all, I have come to realized that it makes zero sense to plot the ASD's of the mode cleaner against the seismometer noise. These measurements are not only quite different, but elementary, they posess different units. I have focused my attention to the MCL being Wiener filtered with the three siesmometer signals. 

One of the major improvements that I make in the following analysis is,

1) Prefiltering; a band pass filter from 1 to 5 Hz, in order to emphasize subtraction of the bump shown in the figure below.

2) I have used vectfit exclusively in the 1 to ~5 Hz range, in order to model the FIR filter properly, as in, the kind of subtraction that we care about. Limiting myself to the 1 - 5 Hz range has allowed me to play freeley with the number of poles, hence being able to fit the FIIR filter properly with an IIR rational transfer function properly,

The resulting ASD's are shown below, in blue we show the raw MCL output, in blac the Wiener filter (FIR) result, and finally in black, the resultant data being filtered with the calculated IIR Wiener filter.

Now, in the following plots I show the IIR Wiener filters for each of the three seismometers, 

X Seismometer,

For the Y seismometer,

and for the Z seismometer,

The matlab code for this work is attached: code.zip

After some surgery yesterday the front ends are all back up and running:

• Eric found that one of the DAC cards in the c1sus front end was not being properly initialized (with the new RCG code).  Turned out that it was an older version DAC, with a daughter board on top of a PCIe board.  We suspected that there was some compatibility issue with that version of the card, so Eric pulled an unused card from c1ioo to replace the one in c1sus.  That worked and now c1sus is running happily.
• Eric put the old DAC card into c1ioo, but it didn't like it and was having trouble booting.  I removed the card and c1ioo came up fine on it's own.
• After all front end were back up and running, all RFM connections were dead.  I tracked this down to the RFM switch being off, because the power cable was not fully seated.  This probably happened when Steve was cleaning up the 1X4/5 racks.  I re-powered the RFM switch and all the RFM connections came back on-line
• All receivers of Dolphin (DIS) "PCIE" IPC signals from c1ioo where throwing errors.  I tracked this down to the Dolphin cable going to c1ioo being plugged in to the wrong port on the c1ioo dolphin card.  I unplugged it and plugged it into the correct port, which of course caused all front end modules using dolphin to crash.  Once I restarted all those models, everything is back:

The interferometer is warming up!

I had some issues locking the IMC at first. It turned out that the MC3 side OSEM signal wasn't getting to the ADC. A satellite box sqush fixed it. 

I touched up the PMC alignment; the best I could do is 0.75V, probably due to the AOM being in place. 

I haven't touched the WFS offsets, but the current ones seem to be doing ok. I'll touch them up tonight when the seismic activity has calmed. 

I made some changes to the state of the PZT/PC crossover gain in the mcdown script, resulting in the IMC catching lock quicker. 

Thankfully, the tip tilt pointing stayed good during the upgrade. I barely had to touch the ETM alignment to lock the arms. ETMX is showing some errant motion, though... 

Bad syntax errors in the c1sus.ini config file were causing the summary pages to crash: a plot type had not been indicated for plots 5 and 6, so I've made these "timeseries."
In the future, please remember to always specify a plot type, e.g.:

BAD:

5 = C1:SUS-ETMX_SUSPIT_INMON.mean,
       C1:SUS-ITMY_SUSPIT_INMON.mean,

GOOD:

3 = C1:SUS-ETMX_SUSPIT_INMON.mean,
       C1:SUS-ITMY_SUSPIT_INMON.mean timeseries

By the way, the pages will continue to be unavailable while I transfer them to the new shared account.

I generated the following plots from the two sets of huddle test data we have for the accelerometers. 

Old data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.

New data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 20 mins worth of data.

I made sure to use the same Impuse response time (6 sec) and sampling frequency (256 Hz), as well as every other parameter for the calculations.

Top left: The resultant self noise curve using the new data, there is definitely and improvement in the 0.5-2 Hz band. 

Top right: Resultant self noise using the old data, for the first set of three accelerometers.

Bottom left: Old data result for the remaining three accelerometers.

Bottom right: Old data result, using all six accelerometers as witnesses instead.

A shared LIGO Data Grid (LDG) account was created for use by the 40m lab. The purpose of this account is to provide access to the LSC computer cluster resources for 40m-specific projects that may benefit from increased computational power and are not linked to any user in particular (e.g. the summary pages).

For further information, please see https://wiki-40m.ligo.caltech.edu/40mLDASaccount

The summary pages are now generated from the new 40m LDAS account. The nodus URL (https://nodus.ligo.caltech.edu:30889/detcharsummary/) is the same and there are no changes to the way the configuration files work. However, the location on LDAS has changed to https://ldas-jobs.ligo.caltech.edu/~40m/summary/ and the config files are no longer version-controlled on the LDAS side (this was redundant, as they are under VCS in nodus).

I have posted a more detailed description of the summary page workflow, as well as instructions to run your own jobs and other technical minutiae, on the wiki: https://wiki-40m.ligo.caltech.edu/DailySummaryHelp

The other night, I spent some time with the mode cleaner. 

First, I made some model changes to the MC_TRANS part of c1mcs.mdl. Specifically, I brought in the userapps QPD part that we are using for the transmon and oplev QPDs. My main motivation for doing so was to have FMs for the pitch and yaw values, to be able to set offsets. Up until now, we have used a QPD centering servo in conjunction with the WFS, but the center of the QPD is not perfectly aligned to represent the center of MC2. Using offsets at the servo isn't really practical, since there are integrators. 

I then spent some time manually aligning the mode cleaner mirrors with WFS off, followed by centering the in-lock MC REFL beam on the WFS QPDs, and setting the WFS and MC_TRANS offsets. (I updated the WFS offset script, and made one for MC_TRANS in scripts/MC/WFS. They now use averaging instead of servoing to zero, a la LSC PD offset script). 

The resultant intracavity power and RIN was an improvement over the older offsets. (RMS RIN went down by half, I think.)

Since Monday, the autolocker seems to be having some trouble. At first, I suspected the changes I made weren't so hot after all, but I've now noticed a pattern. Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh. 

The front panels for the ALS delay line box came in last week. Some of the holes for the screws were slightly misaligned, so I filed those and everything is now put together. I just need to test both front panels to determine if the SMAs should be isolated or not.

Koji had also suggested making the holes in the front and back panel conical recesses so that flat head screws could be used and would counteract the anodization of the panel and avoid the SMAs being isolated. I think if we did that then conductivity would be ensured throughout the panel and also through the rest of the box. I also think one way we could test this before drilling conical recesses would be to test both front panels now, as one has isolated SMAs and one has conductive SMAs. If the anodization of the panel isolated the SMA regardless, we could potentially figure this out by testing both panels. But, would it also be that it is possible that the isolation of the SMA itself does not matter and so this test would tell us nothing? Is there a better way to test if the SMAs are being isolated or not? Or would this be more time consuming than just drilling conical recesses as a preventative measure?

I updated my bandpass filter and have included the bode plot below in Figure 1. It is a fourth order elliptic bandpass filter with a passband ripple of 1dB and a stopband attenuation of 30 dB. It emphasizes the area between 3 and 40 Hz.

Below, I applied this filter to the huddle test data. The results from this were only slightly better in the targeted region than when no pre-filter was applied. 

When I pre-filtered the mode cleaner data and then used an IIR wiener filter, I found that the results did not differ much from the data that was not pre-filtered. I'm not sure yet if I'm targeting the right region of this data with my bandpass filter, and will be looking more into choosing a better region. Also, I am only using certain regions of ff when calculating the transfer function, and need to optimize that region also. I uploaded the code I used to make these plots to github.

In the last post concerning the self noise of the accelerometers, I mentioned the differences between the two data sets I was playing with. In order to give a more concrete analysis of the accelerometers self noise, we came to the conclusion that data taking time should be the same.

The plots below show the analysis for the following two datasets:

Old Data:  6 accelerometers, no cables clamped, no box, 55 mins worth of data.

Newer data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 57.66 mins worth of data, (we had 20 mins of data in the previous data set).

Filter parameters were kept the same in all calculations, the only change that was added to the analysis was the detrending of the signals using the detrend function on Matlab, this improved the results as the plots show. I also plotted the error bars for the Wiener filtered result for reference as well as the manufactures claimed self noise.

After detrending the data and taking a longer dataset we can see the improvement brought upon by the foam box in the low frequency band of 0.5 - 10 Hz, as shown in the bottom left plot. There is a lot of noise that needs to be cancelled out from 10 Hz and on, which brings to our attention the plot on the bottom right corner. This plot uses the old data but uses all six accelerometers as witnesses, it also improved overall after having detrended the data, but what is peculiar about this plot is the fact that it manages to mitigate the higher frequency 10 - ~100 Hz band noise. 

For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?

I've have been talking a little bit with Steve about the seismometer enclosures.

We want to improve on the current stainless steel cans that cover the two Guralps at the end of the arms. In order to do this, we want to cover the interior of the cans with copper foil to improve the thermal conductivity of the enclosure to better control the temperature inside it. Ideally, we would want to copper plate the cans, but cost and difficulty has been an issue.

I have done some rough calculations and it seems that we need a copper layer of thickness being about a third that of the stainless steel can. This happens to be around 0.5-0.6 mm since we have 16 gauge (~1.6 mm) stainless steel cans. 

After wrapping the cans interior with copper, we will insulate them with foam in order to improve its thermal inertia. We want to probably use the same foam that Megan has been using for her seismometer enclosure. I have yet to think about a heater, but something similar to Megans resistor thing would work only smaller. I would be placed inside the can, right on the center of its bottom in order to ditribute heat evenly.

I downloaded new accelerometer huddle test data from last night and analyzed it. This new data set is ~55 mins and uses the same Wiener filter parameters as previous huddle test analysis, the main difference being six accelerometers used in the Wiener filter and the improved experimental set up.

After computing the ASD for the self noise for each of the six accelerometers, (being witnessed by the remaining five), we get,

Now computing the mean of the above signals and the corresponding error bars gives the following result,

Comparing to prevoius huddle tests, I can note two trends on the Wiener subtraction:

1) When using six accelerometers, the subtraction above ~8 Hz drastically improves.

2) When using three accelerometers, there is better cancellation in the 1-5 Hz region, see http://nodus.ligo.caltech.edu:8080/40m/11442. This might have to do with how much more closer the accelerometers are to each other? 

Specially heavy items: old analoge scope or hardware loaded boxes......etc

 The table cover section holding crossbars are not evenly spaced.

You have to center each cover section on the cross bar so it is supported on both sides !

I will clean up on this table tomorrow