I left the arms locked last night. Looks like the drift in the Y arm power is related to the Y arm control signal being much bigger than X.
Also, EQ gave us a better (and not pwd protected) URL for the summary pages. Please replace your previous links with this new one:
Like Steve pointed out, the summary pages show that the y-arm transmission drifts a lot when locked. The OL summary page shows that this is all due to ITMY yaw.
Could be either that they coil driver / DAC is bad or that the suspension is poorly built. We need to dig into ITMY OL trends over long term to see if this is new or now.
Also, weather station needs a reboot. And does anyone know what the MC_F calibration is?
I saw that entry, but it doesn't state what the calibration is in units of Hz/counts. It just gives the final calibrated spectrum.
Still seems to be running without causing FB issues. One thought is that we could look through the FB status channel trends and see if there is some excess of FB problems at 10 min after the hour to see if its causing problems.
I also looked into our minute trend situation. Looks like the files are comrpessed and have checksum enabled. The size changes sometimes, but its roughly 35 MB per hour. So 840 MB per day.
According to the wiper.pl script, its trying to keep the minute-trend directory to below some fixed fraction of the total /frames disk. The comment in the scripts says 0.005%,
but I'm dubious since that's only 13TB*5e-5 = 600 MB, and that would only keep us for a day. Maybe the comment should read 0.5% instead...
The rsync job to sync our frames over to the cluster has been on a 20 MB/s BW limit for awhile now.
Dan Kozak has now set up a cronjob to do this at 10 min after the hour, every hour. Let's see how this goes.
You can find the script and its logfile name by doing 'crontab -l' on nodus.
Reward being offered for the safe return of this thing:
Today Steve and I tried to recenter the Guralps. The breakout box technique didn't work for us, so we just turned the leveling screws until we got the mass position outputs within +/-50 mV for all DoF as read out by the breakout box.
The attachment shows the 6 channels after our work. You can see that GUR2_X/Y still look deadish. I tried wiggling the cables at the interface box and powering on/off, but no luck. Next, we swap cables.
Tried to bring the weather station back to life, but no luck. The unit on the wall is alive and so is the EPICS IOC (c1pem1). But there is apparently no communication between them. telnet into c1pem and the error message repeating at the prompt is:
Weather Monitor Output: NO COMM
Might be related to the flaky connector situation that Liz and I found there a couple summers ago, but I tried jiggling and reseating that one with no luck. Looks like it stopped working around 8 PM on March 24, 2014. That's the same time as a ~30s power outage, so perhaps we just need some more power cycling? Tried hitting the reset button on the VME card for c1pem1, but didn't change anything.
Let's try power cycling that crate (which has c1pem1, c0daqawg, and some GPS receiver)...nope - no luck.
Also tried power cycling the weather box which is near the BS chamber on the wall. This didn't change the error message at the c1pem1 telnet prompt.
The CDSUTILS package has a feature where it substitutes in a C1 or H1 or L1 prefix depending upon what site you are at. The idea is that this should make code portable between LLO and LHO.
Here at the 40m, we have no need to do that, so its better for us to be able to copy and paste channel names directly from MEDM or whatever without having to remove the "C1:" from all over the place.
the way to do this on the command line is (in bash) to type:
To make this easier on us, I have implemented this in our shared .bashrc so that its always the case. This might break some scripts which have been adapted to use the weird CDSUTILS convention, so beware and fix appropriately.
1) Checked the N2 pressures: the unregulated cylinder pressures are both around 1500 PSI. How long until they get to 1000?
2) The IMC has been flaky for a day or so; don't know why. I moved the gains in the autolocker so now the input gain slider to the MC board is 10 dB higher and the output slider is 10 dB lower. This is updated in the mcdown and mcup scripts and both committed to SVN. The trend shows that the MC was wandering away after ~15 minutes of lock, so I suspected the WFS offsets. I ran the offsets script (after flipping the z servo signs and adding 'C1:' prefix). So far powers are good and stable.
3) pianosa was unresponsive and I couldn't ssh to it. I powered it off and then it came back.
4) Noticed that DAQD is restarting once per hour on the hour. Why?
5) Many (but not all) EPICS readbacks are whiting out every several minutes. I remote booted c1susaux since it was one of the victims, but it didn't change any behavior.
6) The ETMX and ITMX have very different bounce mode response: should add to our Vent Todo List. Double checked that the bounce/roll bandstop is on and at the right frequency for the bounce mode. Increased the stopband from 40 to 50 dB to see if that helps.
7) op340 is still running ! The only reason to keep it alive is its crontab:
07 * * * * /opt/rtcds/caltech/c1/burt/autoburt/burt.cron >> /opt/rtcds/caltech/c1/burt/burtcron.log
#46 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo > /cvs/cds/caltech/logs/scripts/FSSslow.cronlog 2>&1
#14,44 * * * * /cvs/cds/caltech/conlog/bin/check_conlogger_and_restart_if_dead
15,45 * * * * /opt/rtcds/caltech/c1/scripts/SUS/rampdown.pl > /dev/null 2>&1
#10 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1
#27 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/RCthermalPID.pl >/cvs/cds/caltech/logs/scripts/RCthermalPID.cronlog 2>&1
00 0 * * * /var/scripts/ntp.sh > /dev/null 2>&1
#00 4 * * * /opt/rtcds/caltech/c1/scripts/RGA/RGAlogger.cron >> /cvs/cds/caltech/users/rward/RGA/RGAcron.out 2>&1
#00 6 * * * /cvs/cds/scripts/backupScripts.pl
00 7 * * * /opt/rtcds/caltech/c1/scripts/AutoUpdate/update_conlog.cron
00 8 * * * /opt/rtcds/caltech/c1/scripts/crontab/backupCrontab
added a new script (scripts/SUS/rampdown.py) which decrements every 30 minutes if needed. Added this to the megatron crontab and commented out the op340m crontab line. IF this works for awhile we can retire our last Solaris machine.
8) To see if we could get rid of the wandering PCDRIVE noise, I looked into the NPRO temperatures: was - T_crystal = 30.89 C, T_diode1 = 21 C, T_diode2 = 22 C. I moved up the crystal temp to 33.0 C, to see if it could make the noise more stable. Then I used the trimpots on the front of the controller to maximimize the laseroutput at these temperatures; it was basically maximized already. Lets see if there's any qualitative difference after a week. I'm attaching the pinout for the DSUB25 diagnostics connector on the back of the box. Aidan is going to help us record this stuff with AcroMag tech so that we can see if there's any correlation with PCDRIVE. The shifts in FSS_SLOW coincident with PCDRIVE noise corresponds to ~100 MHz, so it seems like it could be NPRO related.
Tried swapping cables at the Guralp interface box side. It seems that all of our seismic signal problems have to do with the GUR2 cable being flaky (not surprising since it looks like it was patched with Orange Electrical tape!! rather than proper mechanical strain relief).
After swapping the cables today, the GUR2 DAQ channels all look fine: i.e. GUR1 (the one at the Y end) is fine, as is its cable and the GUR2 analog channels inside the interface box.
OTOH, the GUR1 DAQ channels (which have GUR2 (EX) connected into it) are too small by a factor of ~1000. Seems like that end of the cable will need to be remade. Luckily Jenne is still around this week and can point us to the pinout / instructions. Looks like there could be some shorting inside the backshell, so I've left it disconnected rather than risk damaging the seismometer. We should get a GUR1 style backshell to remake this cable. It might also be possible that the end at the seismometer is bad - Steve was supposed to swap the screws on the granite-aluminum plate on Thursday; I'll double check.
Looking at the summary page trends from today, you can see that the MC transmission is pretty flat after I zeroed the MCWFS offsets. In addition, the transmission from both arms is also flat, indicating that our previous observation of long term drift in the Y arm transmission probably had more to do with bad Y-arm initial alignment than unbalanced ETMY coil-magnets.
Much like checking the N2 pressure, amount of coffee beans, frames backups, etc. we should put MC WFS offset adjustment into our periodic checklist. Would be good to have a reminder system that pings us to check these items and wait for confirmation that we have done so.
Yes - my rampdown.py script correctly ramps down the watchdog thresholds. This replaces the old rampdown.pl Perl script that Rob and Dave Barker wrote.
Unfortunately, cron doesn't correctly inherit the bashrc environment variables so its having trouble running.
On a positive note, I've resurrected the MEDM Screenshot taking cron job, so now this webpage is alive (mostly) and you can check screens from remote:
Too weird. I undid me changes. We'll have to make the C1: stuff work inside each python script.
This makes things act weird:
controls@pianosa|MC 1> z avg 1 "C1:LSC-TRY_OUT"
IFO environment variable not specified.
The c1cal model was maxing out its CPU meter so I logged onto c1lsc and did 'rtcds c1cal stop'. Let's see if this changes any of our FB / DAQD problems.
There's a few hours so far after today's c1cal shut off that the summary page shows no dropouts. I'm not yet sure that this is related, but it seems like a clue.
Good to see that misofw.m is still alive and well. Todo:
Good catch; that was some seriously bad programming on my part. I had some undeclared variable garbage going on. I fixed it and re-implemented the script in CRON on megatron. The log file shows that it has detected no problems for the last several checks. I'll check it again tomorrow to see if its doing good or bad.
I was wondering about the design of the BLRMS fitlers for the seismic channels since the STS ones seem to have so little gain compared to the Guralps.
Here are some plots of the Bode magnitude and impulse responses of the bandpass filters (before the low passing). There's a bunch of entries from Masha on this from her SURF summer. Can anyone comment on why they are all so different?
One of the old Masha entries speaks of designing the lowpass filter in an intelligent way: by adjusting the filter order until the power in the stopband is less than 1% of the power in the passband. Seems like we could do that for bandpass too. For now I have made the names reasonable and changed all of the BP filters to 4th order Butterworth.
Also, it turns out that the Vel2Vel (gain ~0.02) filters were mistakenly on in the STS BP filter banks. The GUR inputs have a gain to scale the counts to velocity, but the STS seem to already be in microns/sec (where is this gain?) so I turned off and deleted the Vel2Vel filters; in any case the gain should not be done seperately in each BP bank, but altogether before the BP filtering.
I think that the real clue was that the dropouts are in some channels and not in others:
As it turns out, the channel with no dropouts is the RAW PSL RMTEMP channel. All the others are the minute trends. So something is up with the trend making or the trend reading in the cluster.
I will move it back. We need to fix our scripts to not use any users/ libraries ever again.
PRM watchdog tripped, but the damprestore.py script wouldn't run.
It turns out the script tries to import some ezca stuff from /users/yuta (), which had been moved to /users/OLD/yuta ().
I've moved the yuta directory back to /users/ until I fix the damprestore script.
Looks like a very handy code, especially with the real statistical tests.
I would make sure to use much smaller excitation amplitudes. Since the coupling is nonlinear, we expect that its only a good noise budget estimator when the excitation amplitude is less than a factor of 3 above the quiesscent excitation.
You are plotting the STS channels, not Guralp. These are for the Trillium 240 seismometer.
Also, you cannot tell if the seismometer is working by plotting the MEAN trend. That just gives the average and we need the fluctuations. Better off looking at the spectrum like I did last time.
And....its not good enough to just do the bubble. You have to do the mass centering procedure that you and I did last time with the breakout paddle.
Additionally, we want to have some set of tests and diagnostics, to make sure we have not introduced unwanted behavior.
To this end I will create some test models and DTT templates, where a series of measurements can be run, like
I'll run these test before touching anything, and make sure I understand all of the results, so that an apples-to-apples comparison can be made after the upgrade is complete.
I got goosebumps just imagining this.
after re-aligning the beam into the PMC, I touched up the steering mirros into the IOO QPDs so that the beams are now centered again. Please don't adjust these references without prior authorization and training.
This plot shows the 10-minute trends for these QPDs over the last 400 days.
To improve our sensor noise estimate, the ACC cables should be clamped using a rubber pad and a big dog clamp on the SP tabletop before exiting the table. Also the sensors themselves should be covered with a foam box or a double cardboard box. The high-frequency acoustic noise may limit the 10 Hz performance since piezos are not very linear.
I've set up the Wilcoxen accelerometers on the SP table for a huddle test, to estimate their noise levels.
They're clamped down to the table along the same axis, and their housings are in good contact, in hopes to make them as correlated as possible.
Wilcoxon. Not Wilconox or Wilco or Wilcoxen. Have some pity on future keyword searchers.
In any case, this signal difference is not big, so we should not need a different amplifier chain for the two signals. The 20 dB of amplification in the BeatBox was a fine way, but not great in circuit layout.
The BBPD has an input referred current noise of 10 pA/rHz and a transimpedance of 2 kOhm, so an output voltage noise of 20 nV/rHz (into 50 Ohms). This would be matched by an Amp with NF = 26 dB, which is way worse than anything we could bur from mini-circuits, so we should definitely NOT use anything like the low-noise, low output power amps used currently (e.g. ZFL-1000LN....never, ever use these for anything). We should use a single ZHL-3A-S (G = 25 dB, NF < 6 dB, Max Out = 30 dBm) for each channel (and nothing else) before driving the cables over to the LSC rack into the aLIGO demod board. I just ordered two of these now.
The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.
Well, green looks better than blue, but it makes the PCDRIVE go high, which means its starting to saturate the EOM drive. So we can't just maximize the phase margin in the PZT/EOM crossover. We have to take into account the EOM drive spectrum and its RMS.
Also, your gain bump seems suspicious. See my TF measurements of the crossover in December. Maybe you were saturating the EOM in your TF ?
Lets find out what's happening with FSS servos over in Bridge and then modify ours to be less unstable.
I made a measurement of the MC2 actuator transfer function by injecting noise from 1-100Hz into LSC_MC2_EXC for about 15 minutes, then estimating the TF from MC2_OUT to IOO_MC_L with CSD/PSD. The inverse of this TF will be applied to their Wiener target data to give us the direct subtration filter we want.
I think what happened here is you forgot to undo the MC_F whitening filter which is the Generic Pentek Interface board next to the MC servo board. I suggest you guys measure this on Monday so you can correctly estimate the MC length noise. And then perhaps undo the whitening in the anti-whitening filter of this filter bank so that the signal which is recorded is in units of kHz.
This should allow your online subtraction filter to be more correct: roughly speaking, the phase shift below a pole or zero is going to be 45*(f/fp) deg. Since we expect there to be 2 zeros at 15 Hz, it would be 9 deg phase shift at 1.5 Hz and limit the subtraction to ~80%.
To help find out if Steve really melted the inside of our precious seismometer, lets hook it up using the handheld seismo wand and see if it produces volts when we shake the ground.
Also, please stop using names like GurA or Gur1 or GurSuzy. We have GurX and GurY because they are at those ends. Anything else is confusing.
Somehow it seems like the ELOG makes all of the thumbnails way too big by default. Did we get some sneaky upgrade recently?
I would only plot your results below 50 Hz. We don't care about the RMS at high frequencies and it can make the RMS misleading.
We definitely need to include one vertical Wilconox at each MC chamber so that it can subtract all of that junk at 10-20 Hz.
I'm not totally sure, but by eyeball, this seems like the best online MCL reduction we've ever had. Nice work.
The 3 Hz performance is the same as usual, but we've never had such good 1 Hz reduction in the online subtraction.
I would like to see a plot of the X & Y arm control signals with only the MCL filter ON/OFF. This would tell us how much of the arm signals were truly frequency noise.
Let's order a pair of 35.5 MHz Wenzel for this guy and package like Rich has done for the WB low noise oscillators.
WE're only sending 6 dBm into it now and its using a 13 dBm mixer. Bad for PMC stability.
Also, if anyone has pix of the servo card, please add them to the DCC page for the PMC.
The IMC often was making that scratchy noise when first catching lock and sometimes breaking. Thinking of the crappy crossover sit that EQ showed in his latest plots, I decided that it didn't make sense to acquire lock with an unstable PZT/EOM crossover, so I have changed mcdown to acquire with +13 dB Fast Gain and its much fast now and no longer makes that sound.
I also changed the caput command from 'caput -l' to 'caput -c -l' to see if the async 'wait for callback' feature will insure that the commands get sent. I witnessed the mcdown not actually writing all of its commands once or twice tonight. With the MC Boost left on its never going to lock.
mcdown has been committed to SVN. Please, if you have recently edit mcup and Autolock, commit them to the SVN or else I will delete them and do an svn up.
ran the ON script several times and it kept pulling it away from good alignment, even when TRX was > 0.9.
Also, for what reason was this model run at 16 kHz?? Makes no sense to me to have a low frequency servo system run so high. Only makes for more digital precision noise, more CPU time, etc. Of course, running it at 2k would mean having to think about all of the AA filtering needed to go up/down from 2k to 16k.
Back in 2011, JoeB wrote some entries on how to automatically update the Simulink webview stuff.
Somehow, the cron broke down over the years. I reran the matlab file by hand today and it worked fine, so now you can see the up to date models using the internet.
I experimented with removing somethings here and there to reduce the c1cal runtime. Eventually I deleted the LSC Sensing Matrix from it.
After removing sensing matrix, the run time is now down to 6 usec.
added the cron script for this to megatron to run at 8:44 AM each morning. Here's the new MegaCron attached :-()-
** it takes ~13 minutes to complete on megatron
# m h dom mon dow command
#0 */1 * * * bash /home/controls/public_html/summary/bin/c1_summary_page.sh > /dev/null 2>&1
#15 5 * * * /ligo/apps/nds2/nds2-megatron/test-restart
# MEDM Screen caps for the webpage
2,13,25,37,49 * * * * /cvs/cds/project/statScreen/bin/cronjob.sh
# op340m transplants -ericq
Since Andrey's SUS Drift mon screen back in 2007, we've had several versions which used different schemes and programming languages. Diego made an update back in January.
Today I added his stuff to the SVN since it was lost in the NFS disks somewhere. Its in SUS/DRIFT_MON/.
Since we've been updating our userapps directory recently to pull in the screens and scripts from the sites, we also got a copy of the Thomas Abbott drift mon stuff which is better (Diego actually removed the yellow/red functionality as part of the 'upgrade'), but more complicated. For now we have the old one. I updated the good values with all optics roughly aligned just a few minutes ago.
Q and Ignacio were taking a second look at the Pentek interface board which we're using to acquire the POP QPD, ALS trans, and MCF/MCL channels. It has a differential intput, two jumper able whitening stages inside and some low pass filtering.
I noticed that each channel has a 1.5 kHz pole associate with each 150:15 whitening stage. It also has 2 2nd order Butterworth low pass at 800 Hz. Also there's a RF filter on the front end. We don't need all that low passing, so I started modifying the filters. Tonight I moved the 800 Hz poles to 8000 Hz. Tomorrow we'll move the others if Steve can find us enough (> 16) 1 nF SMD caps (1206 NPO).
After this those signals ought to have less phase lag and more signal above 1 kHz. Since the ADC is running at 64 kHz, we don't need any analog filtering below 8 kHz.
Nice going. I think the LLO / LHO scheme is to acquire on 1F and then cdsutils avg to get the 3F offsets. The thinking is that that 1F signals have less intrinsic offset than the 3F signals, so we want to be use digital offsets for the 3F locks.
Frustrated by the single pixel width of the windows and how hard that makes it to drag things around, I explored StackExchange:
which showed how there is a .xml file which can be edited to increase this. I've changed the border size to 4 pixels on Rossa - its nice.
The new stage missed the right height by ~2 mm.
Even if I completely bottom out the (New Focus 9071) 4-axis stage, its not short enough. So I removed the AOM from the beam and re-aligned into the PMC.
Steve, please get the aluminum piece remachined to go down by 2.5 mm so we can have some height adjustment room.
New stage can hold the correct polarization.
Also, the turning mirror mount just after the EOM and before the AOM is a U-100 and we want it to be a Suprema for stability - let's not forget to swap that after Steve gets the mount fixed.
As it turned out, the "STS" BLRMS filters were all a mess, so I fixed them up today:
The "C1:PEM-SEIS_STS_1" filter banks are currently empty, so the signal is just in ADC counts. However, by amazing luck, this seems to be the right gain (within a factor of 2) to put the signal into units of microns / second. According the the schematic (D1000749), the default gain of 110 can be switched to make the whole box just have a gain of 2 (differential in, differential out). I wonder if anyone, like Jenne, knows if this is what we have? There's no elog I found about setting the gain switch.
According to the manual, the gain is ~1175 V/(m/s). Our ADC gain should be (2^16)/(40). So:
cal_gain = 1175 * 2 * 65536 / 40 ==>> 0.26 (m/s)/counts
I have put this into the STS_1_X,Y,Z filter modules in c1pem so that these channels are now calibrated. I also put the first few s-domain poles/zeros into the filter based on the manual so that the magnitude in the 10-30 Hz band is correct-ish now.
* Does anyone know how to center the masses on this thing?
I converted our MC WFS relief from CSH to BASH today. I also added 'wait' commands and 'echo' commands so that all DoFs run in parallel nicely. It can be accessed from the MC WFS screen.
I increased the overall MC WFS gain input slider from 0.02 to 0.1 (its in the mcwfson script). The MC Trans loops now have a time constant of ~30 seconds. The relief script relieves ~90% of the MC WFS control signals in the 2 minutes that its allowed to run.
On the next upgrade, we should make it python and have it kill the relief process if the MC loses lock before relief is applied via the alignment sliders.
We looked at the DRMI noise spectrum and chose new excitation frequencies such that the lines are lower in frequency than before (~300 Hz instead of 800 Hz) and also not in some noisy region.
New filters is saved and loaded for all LSC DOFs.
When making the Wiener filter OFF/ON comparisons, we want to use the median PSD estimates, not the mean (which is what pwelch gives you).
cf. Sujan's note and Evan's follow-up
The median will be less sensitive to the transients / gltiches and will show more improvement I think.
I turned on the MCF FF in the OAF today (we need to fix the labeling of the 'ON' buttons on the RHS of the screen). The performance is still good; before / after attached.
Not only is the 1 Hz performance in the MC still good, but the X & Y arm noise reduction is ~1 order of magnitude. Good to know that the filters aren't changing much with time.
Can we just leave this on all the time now? Seems to be OK and there's no visible increase in the arm noise with this on.
Also did some updates to the summary pages and added a CDS FEC tab for CPU times.
Please take a look at the summary pages and bring a list of demands to the Wednesday meeting.