Might have to also get the OL screens that go with this new code to see, but the calibrations don't go into the matrix, but rather into the OLPIT/OLYAW filter banks which follow the division, but before the servo filter banks.
The Schnupp asymmetry is definitely not an important parameter (no need for Koji to explode). It only serves to give us a bigger Q phase signal slope, but is not significant for the I phase signals.
The main anomaly is that the REFL33 optical gain (W/m) seems to have been reduced so much by the phase and amplitude adjustment of the 55 MHz modulation signal. One guess is that the true 3f signal is being made not by the (2*f1 - (-f1)) beat, but by some higher order beat. In addition to the usual RF fields produced by the 2 modulations, we must consider that the sidebands on sidebands produce intermodulation fields just after the EOM and so the fields with which we interrogate the PRMI are more complicated.
Jenne's Optickle calculation today should show us what the sensing matrix contribution is from each pair of fields, so that we can have a sensing matrix signal budget.
I have adapted one of Evan's python scripts into an ipython notebook for calculating our PRMI sensing matrix - the work is ~half done.
The script gets the data from the various PD channels (like REFL33_I) and demdoulates it at the modulation frequencies. At the moment its using just the sensing channels, but with the recent addition of the SUS-LSC_OUT_DQ channels, we can demod the actuation channels as well and not have to hand code the exc amplitudes and the basolute phase. Please ignore the phase for the moment.
The attached PDF shows the demod (including lowpass) outputs for a 2 minute stretch of PRMI locked on f2. Next step is to average these numbers and make the radar plots with the error bars. The script is scripts/LSC/SensingMatrix/PRMIsensMat.ipynb and is in the SVN now.
** along the way, I noticed that the reason this notebook hasn't been working since last night is that someone sadly installed a new anaconda python distro today without telling anyone by ELOG. This new distro didn't have all the packages of the previous one. I've updated it with astropy and uncertainties packages.
I've fixed the Radar plot making part, so that's now included too. The radial direction is linear, so you can see from the smearing of the blobs that the uncertainty is represented in the graphics due to each measurement being a small semi-transparent dot. Next, we'll put the output of the statistics on the plot: mean, std, and kurtosis.
Just in case there was some confusion, the champagne on my desk is not to be opened before I get back, no matter how many signals are transitioned to RF.
In the attached plot you can see that the MC REFL fluctuations started getting larger on Feb 24 just after midnight. Its been bad ever since. What happened that night or the afternoon of Feb 23?
The WFS DC spot positions were far off (~0.9), so I unlocked the IMC and aligned the spots on there using the nearby steering mirrors - lets see if this helps.
Also, these mounts should be improved. Steve, can you please prepare 5 mounts with the Thorlabs BA2 or BA3 base, the 3/4" diameter steel posts, and the Polanski steel mirror mounts? We should replace the mirror mounts for the 1" diameter mirrors during the daytime next week to reduce drift.
This has been done before:
Arm length measurements and g-factor estimates in 2012, but only with an accuracy of ~30 cm. However, Yuta was able to get many FSRs somehow.
MC Refl alignment follow up: the alignment from last night seems still good today. We should keep an on the MC WFS DC spots and not let them get beyond 0.5.
According to the official rules, we only need 8 seconds to declare it "locked".
I wonder if the double cavity pole compensation filter for CARM was on for all the attempts yesterday? IF it looks like it will not saturate, it would be more stable to have the whitening on for REFL11 / AS55. Since on Friday, I set the REFL165 demod phase just by minimizing the MICH control signal with the arms on resonance, we ought to check out the PRMI degeneracy with the ETMs misaligned.
Speaking of signal mixing: Although we weren't able to get the carrier term cancelled in the 3*f1 signals by the relative mod phase method, I wonder if we can do it by mixing the 3*f1 and 3*f2 signals in the input matrix. Might help to keep the PRMI more stable, if that's an issue.
P.S. I have done some scripts directory / SVN cleanup. Adding some directories that were not in (like lockloss) and then removing stuff from the repo using 'svn rm --keep-local filenames' for the image and data files.
Since the DARM_OUT signal is only 500 counts_peak, I don't see why AS55 whitening needs to be switched on. Maybe in a couple weeks after the lock is robust. In any case, its much better to do the switching BEFORE you're using AS55, not after.
Another thought about the mystery PCDRIVE noise: we've been thinking that it could be some slow death inside the NPRO, but it might also be a broken and intermittent thing in the MC servo or MC REFL PD.
Another possibility is that its frequency noise in the old oscillator used to drive the pre-PMC EOM (which is the Pockel Cell for the FSS). To check this, we should swap in a low noise oscillator for the PMC. I have one for testing which has 36 and 37 MHz outputs.
The oplev situation still seems unresolved - notice this DTT. I guess there are still inconsistencies in the screens / models etc.
Could use some some investigation and ELOGGING from Eric.
So, was there real shifting in the ITMX alignment as seen in the DV trend or just mis-diagnosis from the ETMX violin mode? Or how would the ETMX violin mode drive the ITMX with the LSC feedback disabled?
The rsync job to sync our frames over to the cluster has been on a 20 MB/s BW limit for awhile now.
Dan Kozak has now set up a cronjob to do this at 10 min after the hour, every hour. Let's see how this goes.
You can find the script and its logfile name by doing 'crontab -l' on nodus.
This is a very cool result. I'm surprised that it can work so good. Please post what frequency dependent weighting you used on the target before running the Wiener code.
I think its important to tune it to keep the low frequencies from getting amplified by a factor of 10 (as they are in your plots). The seismometers are all just noise acting like thermometers and tilt sensors below 0.1 Hz. Temperature and tilt do not couple to our interferometer very much.
It also seems weird that you would need 20th order filters to make it work good. In any case, you can always split the SOS up into pieces before making the digital filters. For LLO, we used 3 buttons in some cases.
Before locking for the evening, I wanted to try again implementing the Wiener filters that I had designed back in Jaunary (elog 10959).
Unfortunately, this kind of trend plot is not detailed enough to know if something has gone bad in a quantitative way. But at least we can tell that the suspension wire didn't break.
+1 to both Evan and Zach for prompt info and +2 to you for getting more stuff than you started looking for. -2 karma to whomever had swiped them and hoarded without signing. You should put a 40m sticker on both of them. Make sure to check / use fresh batteries. The Busby box is BJT based and works on low impedance sources, the Rai box works on anything, but (I am guessing) has less CMRR.
The Rai box was in the Cryo lab, and the Busby box was in the TCS lab. Neither had been signed out. Lame. Anyhow, thanks to Evan and Zach's memories of having seen them recently, they have been returned to the 40m where they belong. (Also, I grabbed a spare Marconi while I was over there, for the phase noise measurement).
It doesn't work with the lens in there, but it seems pretty close. Please leave it as is and I'll play with it after 5 today.
To test what the inherent angular noise of the HeNe 1103P laser is, we're testing it on a table pointing into the BS OL QPD with only a few steering mirrors.
From the setup that I found today, I've removed the lens nearest to the laser (which was used for the BS and PRM) as well as the ND filter (what was this for?) and the lens placed just before the BS QPD.
With the ND filter removed, the quadrant signals are now ~15000 if we misalign it and ~9000 each with the beam centered.
In order to calibrate the OLPIT_IN1 and OLYAW_IN1 signals into mm of beam motion, I misaligned the mirror just before the QPD. The knobs on there actuate the 100 TPI screws and the knurling on the knob itself has 10 ridges, so that's 36 deg per bump.
PIT cal ~ 1.55 (knob deg / count) -->> 10 microns / count --->>> 10 urad / count
YAW cal ~ 1 (knob deg / count) -->> 6.5 microns / count --->>> 6.5 urad / count
Distance from the 45 deg turning mirror to the QPD silicon surface is 23 cm. Distance between knob tip and fixed pivot point is ~4 cm. 1 knob turn = 0.01" = 0.254 mm = 0.254/40 radians of mirror angle.
So 360 deg of knob gives 2*0.254/40 = 0.012 radians of beam angle = 0.012 * 230 mm ~2.3 mm of beam spot motion. Or 6.4 microns of translation / deg of knob.
The distance from the face of the laser to the QPD is 96 cm.
The punchline is that the laser shows a level of noise which has a similar shape to what's seen at LLO, but 10x lower.
The noise at 0.05 - 0.2 Hz is ~2-3x worse than the PR3 at LLO. Not sure if this is inherent to the HeNe or the wind in our setup.
Just randomly found this old entry from 3 years ago. We should never have installed a GAP 2000 - they are an inferior type of InGaAs diode. We should add to our list replacing these with a 2 mm EG&G diode.
How many 2 mm EG&G InGaAs diodes do we have Steve? Can you please find a good clean diode case so that we can store them in the optics cabinet on the south arm?
We replaced the dead photodiode on MC REFL PD with a new one (GAP 2000). We measured the frequency response of the PD and tuned the resonant frequency using inductor L5 (in the circuit diagram) to be 29.575MHz - over an average of 10 measurements.
Q: please update this Wiki page with the go-back procedure:
For 1/4-20 bolts made of 18-8 Stainless Steel, the recommended torque varies from 65-100 inch-pounds, depending upon the application, the lubrication, how loose the bolt is, if there's a washer, etc.
For our case, where we are going into a tapped, ferromagnetic stainless table, its less clear, but it will certainly by in the 60-80 range. This is close to the 5-6 foot-lbs that I recommended on Wednesday.
I've ordered 3 torque wrenches with 1/4" drive so that we can have one at each end and one in the toolbox near MC2. We'll indicate the recommended torque on there so that we can tighten everything appropriately.
Recently, Steve replaced the HeNe which was sourcing the BS & PRM OL. After replacement, no one checked the beam sizes and we've been living with a mostly broken BS OL. The beam spot on the QPD was so tiny that we were seeing the 'beam is nearly the size of the segment gap' effect.
Today I removed 2 of the lenses which were in the beam path: one removed from the common PRM/BS path, and one removed from the PRM path. The beams on both the BS & PRM got bigger. The BS beam is bigger by a factor of 7. I've increased the loop gains by a factor of 6 and now the UGFs are ~6 Hz. The loop gains were much too high with the small beam spots that Steve had left there. I would prefer for the beams to be ~1.5-2x smaller than they are now, but its not terrible.
Many of the mounts on the table are low quality and not constructed stably. One of the PRM turning mirror mounts twisted all the way around when I tried to align it. This table needs some help this summer.
In the future: never try locking after an OL laser change. Always redo the telescope and alignment and check the servo shape before the OL job is done.
Also, I reduced the height of the RG3.3 in the OL loops from 30 to 18 dB. The BS OL loops were conditionally stable before and thats a no-no. It makes it oscillate if it saturates.
Upgraded python on megatron. Added lines to the crontab to run autoMX.py. Edited crontab to have a PYTHONPATH so that it can run .py stuff.
But autoMX.py is still not working from inside of cron, just from command line.
Since python from crontab seemed intractable, I replaced autoMX.py with a soft link that points at autoMX.sh.
This is a simple BASH script that looks at the LSC FB stat (C1:DAQ-DC0_C1LSC_STATUS), and runs the restart mxstream script if its non-zero.
So far its run 5 times successfully. I guess this is good enough for now. Later on, someone ought to make it loop over other FE, but this ought to catch 99% of the FB issues.
If it all possible, don't use links to your home directory. Its not robust. It would be like if you clicked on your Google Music and it told you to ask me to sing that song to you. Imagine that on auto-repeat next time your fancy-bone itches.
Since the nodus upgrade, Eric/Diego changed the old csh restart procedures to be more UNIX standard. The instructions are in the wiki.
After doing some software updates on nodus today, apache and elogd didn't come back OK. Maybe because of some race condition, elog tried to start but didn't get apache. Apache couldn't start because it found that someone was already binding the ELOGD port. So I killed ELOGD several times (because it kept trying to respawn). Once it stopped trying to come back I could restart Apache using the Wiki instructions. But the instructions didn't work for ELOGD, so I had to restart that using the usual .csh script way that we used to use.
Today I tried some things, but basically, lowering the input gain by 10 made the thing stable. In the attached screenshotstrip, you can see what happens with the gain at 1. After a few cycles of oscillation, I turned the gain back to 0.1.
There still is an uncontrolled DoF, but I that's just the way it is since we only have one mirror (the BS) to steer into the x arm once the yarm pointing is fixed.
Along the way, I also changed the phase for POX, just in case that was an issue. I changed it from +86 to +101 deg. The attached spectra shows how that lowered the POX_Q noise.
I also changed the frequencies for ETM_P/Y dither from ~14/18 Hz to 11.31/14.13 Hz. This seemed to make no difference, but since the TR and PO signals were quieter there I left it like that.
This is probably OK for now and we can tune up the matrix by measuring some sensing matrix stuff again later.
We want to have a VAC page in the summaries, so Steve - please put a list of important channel names for the vacuum system into the elog so that we can start monitoring for trouble.
Also, anyone that has any ideas can feel free to just add a comment to the summary pages DisQus comment section with the 40m shared account or make your own account.
Installed libmotif3 and libmotif4 on nodus so that we can run dataviewer on there.
Also, the lscsoft stuff wasn't installed for apt-get, so I did so following the instructions on the DASWG website:
Then I installed libmetaio1, libfftw3-3. Now, rather than complain about missing librarries, diaggui just silently dies.
Then I noticed that the awggui error message tells us to use 'ssh -Y' instead of 'ssh -X'. Using that I could run DTT on nodus from my office.
Still processing, but I think it should work fine once we have a day of data. Until then, here's the summary pages so far, including Vac channels:
Same thing again today. So I renamed the /etc/init/elog.conf so that it doesn't keep respawning bootlessly. Until then restart elog using the start script in /cvs/cds/caltech/elog/ as usual.
I'll let EQ debug when he gets back - probably we need to pause the elog respawn so that it waits until nodus is up for a few minutes before starting.
Looks like some of our seismometers are oscillating, not mounted well, or something like that. No reason for them to be so different.
Which Guralp is where? And where are our accelerometers mounted?
X arm was far out in yaw, so I reran the ASS for Y and then X. Ran OK; the offload from ASS outputs to SUS bias is still pretty violent - needs smoother ramping.
After this I recentered the ITMX OL- it was off by 50 microradians in pitch. Just like the BS/PRM OLs, this one has a few badly assembled & flimsly mounts. Steve, please prepare for replacing the ITMX OL mirror mounts with the proper base/post/Polaris combo. I think we need ~3 of them. Pit/yaw loop measurements attached.
Based on the PEM-SEIS summary page, it looked like GUR1 was oscillating (and thereby saturating and suppressing the Z channel). So I power cycled both Guralps by turning off the interface box for ~30 seconds and the powering back on. Still not fixed; looks like the oscillations at 110 and 520 Hz have moved but GUR2_X/Y are suppressed above 1 Hz, and GUR1_Z is suppressed below 1 Hz. We need Jenne or Zach to come and use the Gur Paddle on these things to make them OK.
From the SUS-WatchDog summary page, it looked like the PRM tripped during the little 3.8 EQ at 4AM, so I un-tripped it.
Caryn's temperature sensors look like they're still plugged in. Does anyone know where they're connected?
I left the arms locked last night. Looks like the drift in the Y arm power is related to the Y arm control signal being much bigger than X.
Also, EQ gave us a better (and not pwd protected) URL for the summary pages. Please replace your previous links with this new one:
Like Steve pointed out, the summary pages show that the y-arm transmission drifts a lot when locked. The OL summary page shows that this is all due to ITMY yaw.
Could be either that they coil driver / DAC is bad or that the suspension is poorly built. We need to dig into ITMY OL trends over long term to see if this is new or now.
Also, weather station needs a reboot. And does anyone know what the MC_F calibration is?
I saw that entry, but it doesn't state what the calibration is in units of Hz/counts. It just gives the final calibrated spectrum.
Still seems to be running without causing FB issues. One thought is that we could look through the FB status channel trends and see if there is some excess of FB problems at 10 min after the hour to see if its causing problems.
I also looked into our minute trend situation. Looks like the files are comrpessed and have checksum enabled. The size changes sometimes, but its roughly 35 MB per hour. So 840 MB per day.
According to the wiper.pl script, its trying to keep the minute-trend directory to below some fixed fraction of the total /frames disk. The comment in the scripts says 0.005%,
but I'm dubious since that's only 13TB*5e-5 = 600 MB, and that would only keep us for a day. Maybe the comment should read 0.5% instead...
Reward being offered for the safe return of this thing:
Today Steve and I tried to recenter the Guralps. The breakout box technique didn't work for us, so we just turned the leveling screws until we got the mass position outputs within +/-50 mV for all DoF as read out by the breakout box.
The attachment shows the 6 channels after our work. You can see that GUR2_X/Y still look deadish. I tried wiggling the cables at the interface box and powering on/off, but no luck. Next, we swap cables.
Tried to bring the weather station back to life, but no luck. The unit on the wall is alive and so is the EPICS IOC (c1pem1). But there is apparently no communication between them. telnet into c1pem and the error message repeating at the prompt is:
Weather Monitor Output: NO COMM
Might be related to the flaky connector situation that Liz and I found there a couple summers ago, but I tried jiggling and reseating that one with no luck. Looks like it stopped working around 8 PM on March 24, 2014. That's the same time as a ~30s power outage, so perhaps we just need some more power cycling? Tried hitting the reset button on the VME card for c1pem1, but didn't change anything.
Let's try power cycling that crate (which has c1pem1, c0daqawg, and some GPS receiver)...nope - no luck.
Also tried power cycling the weather box which is near the BS chamber on the wall. This didn't change the error message at the c1pem1 telnet prompt.
The CDSUTILS package has a feature where it substitutes in a C1 or H1 or L1 prefix depending upon what site you are at. The idea is that this should make code portable between LLO and LHO.
Here at the 40m, we have no need to do that, so its better for us to be able to copy and paste channel names directly from MEDM or whatever without having to remove the "C1:" from all over the place.
the way to do this on the command line is (in bash) to type:
To make this easier on us, I have implemented this in our shared .bashrc so that its always the case. This might break some scripts which have been adapted to use the weird CDSUTILS convention, so beware and fix appropriately.
1) Checked the N2 pressures: the unregulated cylinder pressures are both around 1500 PSI. How long until they get to 1000?
2) The IMC has been flaky for a day or so; don't know why. I moved the gains in the autolocker so now the input gain slider to the MC board is 10 dB higher and the output slider is 10 dB lower. This is updated in the mcdown and mcup scripts and both committed to SVN. The trend shows that the MC was wandering away after ~15 minutes of lock, so I suspected the WFS offsets. I ran the offsets script (after flipping the z servo signs and adding 'C1:' prefix). So far powers are good and stable.
3) pianosa was unresponsive and I couldn't ssh to it. I powered it off and then it came back.
4) Noticed that DAQD is restarting once per hour on the hour. Why?
5) Many (but not all) EPICS readbacks are whiting out every several minutes. I remote booted c1susaux since it was one of the victims, but it didn't change any behavior.
6) The ETMX and ITMX have very different bounce mode response: should add to our Vent Todo List. Double checked that the bounce/roll bandstop is on and at the right frequency for the bounce mode. Increased the stopband from 40 to 50 dB to see if that helps.
7) op340 is still running ! The only reason to keep it alive is its crontab:
07 * * * * /opt/rtcds/caltech/c1/burt/autoburt/burt.cron >> /opt/rtcds/caltech/c1/burt/burtcron.log
#46 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo > /cvs/cds/caltech/logs/scripts/FSSslow.cronlog 2>&1
#14,44 * * * * /cvs/cds/caltech/conlog/bin/check_conlogger_and_restart_if_dead
15,45 * * * * /opt/rtcds/caltech/c1/scripts/SUS/rampdown.pl > /dev/null 2>&1
#10 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1
#27 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/RCthermalPID.pl >/cvs/cds/caltech/logs/scripts/RCthermalPID.cronlog 2>&1
00 0 * * * /var/scripts/ntp.sh > /dev/null 2>&1
#00 4 * * * /opt/rtcds/caltech/c1/scripts/RGA/RGAlogger.cron >> /cvs/cds/caltech/users/rward/RGA/RGAcron.out 2>&1
#00 6 * * * /cvs/cds/scripts/backupScripts.pl
00 7 * * * /opt/rtcds/caltech/c1/scripts/AutoUpdate/update_conlog.cron
00 8 * * * /opt/rtcds/caltech/c1/scripts/crontab/backupCrontab
added a new script (scripts/SUS/rampdown.py) which decrements every 30 minutes if needed. Added this to the megatron crontab and commented out the op340m crontab line. IF this works for awhile we can retire our last Solaris machine.
8) To see if we could get rid of the wandering PCDRIVE noise, I looked into the NPRO temperatures: was - T_crystal = 30.89 C, T_diode1 = 21 C, T_diode2 = 22 C. I moved up the crystal temp to 33.0 C, to see if it could make the noise more stable. Then I used the trimpots on the front of the controller to maximimize the laseroutput at these temperatures; it was basically maximized already. Lets see if there's any qualitative difference after a week. I'm attaching the pinout for the DSUB25 diagnostics connector on the back of the box. Aidan is going to help us record this stuff with AcroMag tech so that we can see if there's any correlation with PCDRIVE. The shifts in FSS_SLOW coincident with PCDRIVE noise corresponds to ~100 MHz, so it seems like it could be NPRO related.
Tried swapping cables at the Guralp interface box side. It seems that all of our seismic signal problems have to do with the GUR2 cable being flaky (not surprising since it looks like it was patched with Orange Electrical tape!! rather than proper mechanical strain relief).
After swapping the cables today, the GUR2 DAQ channels all look fine: i.e. GUR1 (the one at the Y end) is fine, as is its cable and the GUR2 analog channels inside the interface box.
OTOH, the GUR1 DAQ channels (which have GUR2 (EX) connected into it) are too small by a factor of ~1000. Seems like that end of the cable will need to be remade. Luckily Jenne is still around this week and can point us to the pinout / instructions. Looks like there could be some shorting inside the backshell, so I've left it disconnected rather than risk damaging the seismometer. We should get a GUR1 style backshell to remake this cable. It might also be possible that the end at the seismometer is bad - Steve was supposed to swap the screws on the granite-aluminum plate on Thursday; I'll double check.
Looking at the summary page trends from today, you can see that the MC transmission is pretty flat after I zeroed the MCWFS offsets. In addition, the transmission from both arms is also flat, indicating that our previous observation of long term drift in the Y arm transmission probably had more to do with bad Y-arm initial alignment than unbalanced ETMY coil-magnets.
Much like checking the N2 pressure, amount of coffee beans, frames backups, etc. we should put MC WFS offset adjustment into our periodic checklist. Would be good to have a reminder system that pings us to check these items and wait for confirmation that we have done so.
Yes - my rampdown.py script correctly ramps down the watchdog thresholds. This replaces the old rampdown.pl Perl script that Rob and Dave Barker wrote.
Unfortunately, cron doesn't correctly inherit the bashrc environment variables so its having trouble running.
On a positive note, I've resurrected the MEDM Screenshot taking cron job, so now this webpage is alive (mostly) and you can check screens from remote:
Too weird. I undid me changes. We'll have to make the C1: stuff work inside each python script.
This makes things act weird:
controls@pianosa|MC 1> z avg 1 "C1:LSC-TRY_OUT"
IFO environment variable not specified.
The c1cal model was maxing out its CPU meter so I logged onto c1lsc and did 'rtcds c1cal stop'. Let's see if this changes any of our FB / DAQD problems.
There's a few hours so far after today's c1cal shut off that the summary page shows no dropouts. I'm not yet sure that this is related, but it seems like a clue.