Once you install a matlab newer than 2012a, you can install ligoDV as a matlab app and get the NDS2 client software for free. So you can easily get the 40m data from the outside world now and do the analysis on your own computer rather than login through nodus.
In the past, we used to use Stefan's 'ezcademod' or Matt's 'ezlockin' to do auto phase adjustment.
JoeB / Jamie are working on python replacements for these tools, but in the near term possibly I can make a bash script to use ezcaservo and the existing LOCKINs to do this.
I took the "aso-laptop" and made it into Ubuntu a couple months ago. Today I added it to the Martian network and then moved it to the X End.
I followed the instructions in (https://wiki-40m.ligo.caltech.edu/Network) and added it to the files in /var/named/chroot/var/named on linux1 and did the "service named restart".
The router already had his MAC address in its list (because Yoichi was illegally using his personal laptop on the Martian). The new laptop's name is 'asia'. This is a legal name according to our computer naming conventions and this Wikipedia page (http://en.wiktionary.org/wiki/Category:Italian_female_given_names). It has been added to the Name Pool on the wiki.
The terminal on the laptop still calls itself 'aso-laptop' so I need some help in fixing that. It successfully connects to 40MARS and displays a MEDM sitemap after sshing in to pianosa.
I use 'ssh -X -C' since I find that compression actually helps when the laptops are so far from the router.
Sun Aug 18 15:52:50 2013
Found the FB lights (C1:FEC-NN_FB_NET_STATUS and C1:DAQ-DC0_C1XXX_STATUS) RED for everything on the CDS_FE_STATUS screen.
I used the (! mxstream restart) button ro restart the mxstreams. Everything is green now.
PMC was out of lock- relocked it and the IMC locked itself as did the X & Y arms on IR. X was already green locked.
I noticed at LLO (?) that the LSC screen there uses up ~25-30% of the CPU time on a single core for the control room iMac workstations - this seems excessive.
Here is an accounting of CPU usage percentages for some of our screens:
These were measured using the program 'glances' on rosalba. MEDM running with only the sitemap used up 0.9% of a CPU. With the screens running, the fluctuation from sample to sample could be ~ +/- 0.5%. While the LSC screen seems to be the biggest pig, it is only big in comparison to small pigs. Certainly this pig has gotten bigger after getting sent to Louisiana.
JoeB and JamieR are working somewhat coherently on a set of python libraries to fulfill all of our command line CDS wants. This is being done mostly to satisfy The Guardian and the SkunkTools project.
I did an 'svn up' in /opt/rtcds/userapps (it might finish in ~1000 years) to get the things that they have so far (in particular, Joe's 'pyavg'). There's going to be some issues since the pylib stuff written by Yuta/Kiwamu has never been integrated with anything and is imported as 'epics' in many python scripts. As we move over to the new stuff there will be a lot of broken script functions since the new libraries are also used in that way.
While Jenne was plotting, I locked and aligned the MICH with AS55_Q. Then I aligned the PRM and locked PRMI using REFL55_I/Q with triggering on POP22, but no power normalization.
I used this to set the phase for REFL11 and REFL55 (driving PRM at 111.3 Hz and minimizing the Q response using the DTT Sine Response tool). I flipped the sign on REFL11 by
The REFL11 gain is ~50x larger than REFL55; this is with the 15 dB whitening gain on REFL55 and none for REFL11. What's going on here? The attached PDF shows the two time series with the free swinging PRMI and both phases set to ~ +/- 2 deg. The REFL55 signals have been scaled up by 50x.
So then we went in and looked at the RF signals at the demod boards. To do this we disconnected the RFPD test cables and hooked the RF Mon outputs into the 50 Ohm inputs on a scope. The following PNG images show the scope traces. The REFL11 (yellow) traces are too big!! See how small the REFL55 (green) are. REFL11 is saturating - need to fix.
/home/cds is >98% full - below are some of the usage numbers:
controls@rosalba:/users/OLD 0$ du -h --max-depth=1
controls@rosalba:/opt/rtcds/userapps 0$ du -h --max-depth=1
linux1:cds>nice du -h --max-depth=1
du: `./llo/chans/daq/archive': Permission denied
du: `./llo/chans/daq/old': Permission denied
One of the reasons that our disk is getting full is due to the scripts_archive directory. A backup script runs on op340m and makes a tar.bz2 file of the scripts directory and puts it in scripts_archive every morning at 6 AM.
On Oct 7, 2011, Koji fixed this script to point at our new scripts directory instead of the old /cvs/cds/caltech/scripts directory. Since then, however, no one has fixed the exclude file to NOT back up the junk that's in that directory. Its a 1.6 GB directory so its full of it.
I've deleted a bunch of junk from the scripts directory: this directory is for scripts, not for your personal home movies or junk data files. Put those in your USER directory. Put temporary data files in /tmp/. I've also added a few more patterns to the exclude file so that less .mpg, .png, .pdf, .dat, etc get stored every day. The new daily .tar.bz2 file wil be ~25 MB instead of 770 MB.
(also fixed the backup script to use 'env' to setup the perl environment and removed the hard-coded path to tar)
Yes, this was not ELOG'd by me, unfortunately. This was the MC tickler which I described to some people in the control room when I turned it on.
As Koji points out, with the MCL path turned off this injects frequency noise and pointing fluctuations into the MC. With the MCL path back on it would have very small effect. After the pumpdown we can turn it back on and have it disabled after lock is acquired. Unfortunately, our LOCKIN modules don't have a ramp available for the excitation and so this will produce some transients (or perhaps we can ezcastep it for now). Eventually, we will modify this CDS part so that we can ramp the sine wave.
I've written a new TICKLE script using the newly found 'cavget' and 'cavput' programs. They are in the standard epics distribution as extension binaries. They allow multichannel read/write as well as ramping, delays, incremental steps, etc. http://www.aps.anl.gov/epics/tech-talk/2012/msg01465.php.
Running from the command line, they seem to work fine, but I've left it OFF for now. I'll switch it into the MC autolocker at some point soon.
Meh. 600 counts is too weak. You should fix the electronics so that the maximized green laser transmission gives more like ~10000 counts.
Just to rephrase somewhat:
We can put our scripts for the MICH, PRMI, and DRMI into the IFO CONFIGURE screens for now and then it should be easy to get them into the Guardian once Jamie has the bugs worked out.
This screen can also be used to setup and start the dither alignment for each configuration (once we have one working for DRMI / SRM).
Also, now that the notches/bandstop filters for the violin modes have been move from the SUS into the LSC, we should fix the triggering to engage them a few seconds after the boosts.
I have modified one of the spare demod boards that was sitting above the electronics bench (the one which was unlabeled - the others say 33MHz, 55MHz and 165MHz) to be the new AS110 demod board. In place of the T1 coil, and the C3 and C6 resistors, I have put the commercial splitter PSCQ-2-120+. In place of U5 (the low pass for the PD input) I have put an SCLF-135+.
OK, but what kind of filter should we be actually using? i.e. what purpose the 135 MHz low pass serve in contrast to a PHP-100+ ?
While we were trying to relock the MC after Jenne put back the RF box, we found there was some mysterious motion in MC2. After spending time trying to figure out where this was coming from, the source was found to be at LOCKIN2 of MC2 suspension "The MC TICKLER" that was left enabled. This was turned OFF and MC locked just fine after that.
EDIT JCD: The Tickler should be disabled, if the autolocker is disabled.
Sounds like this was just incidental, since the MC locked fine also with the tickler enabled for weeks.
The tickle is disabled by the down script, but there's no way to correctly handle all possible button pushes. If you want to disable the autolocker for some reason you should run mcdown before trying to lock. This will set up things with the correct settings.
You're right - down turns it on. Still, the fact that the same tickle recently causes a problem and didn't make 20% power fluctuations until now tells me that its not that the tickle amplitude is too large. Whatever changed recently is the problem.
There doesn't seem to be any coherence among the different directions of ground motion (as expected from seismic theory), so I am suspicious of such a low MICH noise.
controls@rosalba:~ 0$ cdsutilsTraceback (most recent call last): File "/ligo/apps/cdsutils/lib/cdsutils/__main__.py", line 7, in <module> from cdsutils import CMDS File "/ligo/apps/cdsutils/lib/cdsutils/__init__.py", line 4, in <module> from servo import servo File "/ligo/apps/cdsutils/lib/cdsutils/servo.py", line 1, in <module> from epics import PVImportError: No module named epicscontrols@rosalba:~ 1$
Mon Sep 16 19:40:32 2013
In May of 2013 Den wrote a PMC Autolocker because he ignored / didn't want to read anyone else's code. Later that year Yuta also wrote another one from scratch for the same reasons.
I tried to use both today, but neither one runs. Yuta's one doesn't run because he was using a bunch of private yuta library stuff in the yuta directory. That kind of programming style is pretty useless for us since it never works after some time.
So I re-activated and tested the PMCAutolock bash script (it is actually a symbolic link called "PMCAutolock" which points to AutoLock.sh). These scripts are all basically the same:
They turn off the loop (or turn down the gain) and then scan the PZT, look for a resonance, and then activate the loop.
One problem with the logic has been that turning off the loop makes the gain so low that the peak flashes by too fast. But leaving the loop ON and just sweeping with the gain turned down to -10 dB is also not good. That only reduces the UGF from 1 kHz to ~100 Hz. What we want is more like a 10 Hz UGF while scanning the length. SO, I edited the script to turn down the modulation depth on the EOM by that factor. After acquiring lock, it returns all settings to the nominal levels as defined on the PSL_SETTINGS screen.
I also changed the .bashrc aliases for the MEDM command so that if you type medm_good at the command line you get MEDM screens with scalable fonts. So you can stretch the screens.
I used a script (~PSL/PMC/testAutoLocker.sh) to unlock the PMC and run autlocker ~100 times to see how robust the new autlocker is.
It failed to grab it 2 out of 137 times. During those times it just went on trying to ramp the PZT even after it had gone to a rail. Once someone resurrects Rob's 'trianglewave' script we should be OK. Even so, I think this is good enough. Please try this out via the yellow button next time the PMC needs to be locked.
It usually takes 10-30 seconds to lock, depending upon where the fringe is compared to the upper voltage rail. Good enough.
OUr disk was getting full again. Turned out my "fix" to 25 MB was only a fix to 250 MB. Since we were getting disk full warnings on our Ubuntu workstations, I deleted some COMSOL.dmg files from users/zach/ and then started deleting every other tarball from the scripts_archive directory. ~221 GB are now free. Still need to fix the exclude file for scripts better.
I used our procedure from this entry to set the IMC board offset as well as the FSS board offset.
I found this afternoon that the MC was having trouble locking: the PC path was railing as soon as the boost was engaged. Could be that there's some misalignment on the PSL which has led to some RAM having to be canceled by this new offset. Let's see if its stable for awhile.
Today I noticed that there was a lot of noise at the Bounce and Roll eigenfrequencies for ETMY. I found that the bandstop filter were set at completely the wrong frequencies, so I've remade them.
The filters were last tuned by Leo in May of 2011. Even so, he left the frequencies at the frequencies of the old MOS suspensions which had f_bounce ~ 12 Hz.
The FOTON plot shows the OLD ones versus the NEW ones. The DTT spectra shows the oplev error signals in the usual state. I have also copied these over to the SUSPOS,PIT,YAW, and SIDE filter banks and turned them all ON.
I also turned OFF and deleted the 3 Hz RG filter that was there. There's no such peak in the error signal and even if one wanted to compensate for the stack mode, it should be a low Q filter, not this monster.
controls@rosalba:/opt/rtcds/caltech/c1/scripts/SUS 0$ ./setOLtramps
Old : C1:SUS-ETMX_OLPIT_TRAMP 0
New : C1:SUS-ETMX_OLPIT_TRAMP 2
Old : C1:SUS-ETMX_OLYAW_TRAMP 0
New : C1:SUS-ETMX_OLYAW_TRAMP 2
Old : C1:SUS-ETMY_OLPIT_TRAMP 2
New : C1:SUS-ETMY_OLPIT_TRAMP 2
Old : C1:SUS-ETMY_OLYAW_TRAMP 2
New : C1:SUS-ETMY_OLYAW_TRAMP 2
Old : C1:SUS-ITMX_OLPIT_TRAMP 0
New : C1:SUS-ITMX_OLPIT_TRAMP 2
Old : C1:SUS-ITMX_OLYAW_TRAMP 0
New : C1:SUS-ITMX_OLYAW_TRAMP 2
Old : C1:SUS-ITMY_OLPIT_TRAMP 0
New : C1:SUS-ITMY_OLPIT_TRAMP 2
Old : C1:SUS-ITMY_OLYAW_TRAMP 0
New : C1:SUS-ITMY_OLYAW_TRAMP 2
Old : C1:SUS-BS_OLPIT_TRAMP 0
New : C1:SUS-BS_OLPIT_TRAMP 2
Old : C1:SUS-BS_OLYAW_TRAMP 0
New : C1:SUS-BS_OLYAW_TRAMP 2
Old : C1:SUS-PRM_OLPIT_TRAMP 0
New : C1:SUS-PRM_OLPIT_TRAMP 2
Old : C1:SUS-PRM_OLYAW_TRAMP 0
New : C1:SUS-PRM_OLYAW_TRAMP 2
Old : C1:SUS-SRM_OLPIT_TRAMP 0
New : C1:SUS-SRM_OLPIT_TRAMP 2
Old : C1:SUS-SRM_OLYAW_TRAMP 0
New : C1:SUS-SRM_OLYAW_TRAMP 2
Done setting TRAMPs
The ETMX oplev signal looks kind of dead compared to the ETMY. It has no features in the spectra and the SUM is pretty low.
I noticed that the cal fields are still set to 1. To get it close to something reasonable, I calibrated it vs. the SUSPIT and SUSYAW values by giving it a step in angle and using 'tdsavg' plus some arithmetic.
OLPIT = 45 urads/ count
OLYAW = 85 urads / count
These are very rough. I don't even know what the accuracy is on the OSEM based calibration, so this ought to be redone in the way that Jenne and Gabriele did before.
The attached image shows the situation after "calibration" of ETMX. This OL system needs some noise investigation.
Having trouble again, starting around 1 hour ago. No one in the VEA. Adjusted the offset -seems to be OK again.
After seeing all of these spikes in the BLRMS at high frequency for awhile, I power cycled the Guralp interface box (@ 10:21 PM) to see if it would randomly recenter in a different place and stop glitching.
It did - needs to be better centered (using the paddle). Plot shows how the Z channel gets better after power cycle.
After relocking the PMC at a good voltage, Steve and I re-aligned the beam into the PMC by walking the last two steering mirrors. After maximizing the power, we also aligned the reflected beam by maximizing the PMC_REFL_DC with the unlocked beam.
Transmission is back to 0.84 V. We need Valera mode matching maintenance to get higher I guess. Maybe we can get a little toaster to keep the PMC PZT more in the middle of its range?
its an acquired taste, but its a must since we're sending an interferometer to India
I went down to investigate the issue with the extra noise that I found in the ETMY optical lever yesterday. There were several problems with the optical layout down there - I'm not sure if I remember them all now.
The main noise issue, however, appears to be not a layout issue at all. Instead its that the laser intensity noise has gone through the roof. See attached spectra of the quadrants (this is the way to diagnose this issue).
I'll ask Steve to either heal this laser or swap it out tomorrow. After that's resolved we'll need another round of layout fixing. I've done a couple of hours today, but if we want a less useless and noisy servo we'll have to do better.
NOTE: by looking at the OL quadrants, I've found a noisy laser, but this still doesn't explain the excess noise in the ETMX. That was the one that has a noisier error signal, not ETMY. By the coherence in the DTT, you can see that the ETMY OL is correctly subtracting and normalizing out the intensity noise of the laser. Seems like the ETMX electronics might be the culprit down there.
Not so fast! We need to plan ahead of time so that we don't have to repeat this ETMY layout another dozen times. Please don't make any changes yet to the OL layout.
Its not enough to change the optics if we don't retune the loop. Please do buy a couple of JDSU (and then we need to measure their intensity noise as you did before) and the 633 nm optics for the mode matching and then we can plan about the layout.
Hidden in Nakano-kun's previous entries was that the phase margin of the X-Arm was only 9 degrees!! This extremely close to instability and makes for huge gain peaking. The feedback loop is increasing noise above 100 Hz rather than suppress. After some tweaks of the LSC filters we got a much more stable loop/.
So we today started to examine the sources of phase lag in the arm cavity sweeps. There were a few unfortunate choices in the XARM LSC filter bank which we tuned to get less delay.
Then I wrote a bunch of detail about how that worked, but the ELOG ate my entry because it couldn't handle converting my error signal noise plot into a thumbnail. Then it crashed and I restarted it. We also have now propagated the changes to the Y arm by copy/paste the filters and the result there is pretty much the same: low phase margin is now 38 deg phase margin. Noise is less bad.
I fixed the XARM and YARM real time calibration servo.
I also change the C1CAL_MICH_A servo. Now the actuator response and the suspension TF are combined together and that filter name is BS_act. C1CAL_XARM_A and C1CAL_YARM_A have same kind of filters, ETMX_act and ETMY_act.
There are AI filter in each A servo and inv_AA, inv_DAA filters in CINV servo, but it's doesn't work correctly yet.
These aren't servos. What he means is that he's changed some filters in the real time calibration screens so as to make the actuation and sensing parts more accurate, but the inversion of the AA filters is not accurate yet.
I think we can use the IMC autolocker to start with getting this started. Once Jamie fixes the NDSSERVER environment variable bug, we should be able to use his more slick automation code to make it auto lock.
Probably more important is to establish quantitatively how particle counts affects the lock acquisition or noise in the interferometer.
We don't want to adopt a "Sky is Falling" mentality as was done previously in LIGO when people were trying to outlaw burritos and perfume.
Dust monitor counts and human noses do not correlate well with the interferometer's nose.
When the vacuum system is closed, we might check to see if particle counts correlate with losses in the PMC or excess scatter on the ISC tables. If not, we should move on to other concerns.
Yesterday and this morning's slow NFS disk access was caused by 'svndumpfilter' being run at linux1 to carve out the Noise Budget directory. It is being moved to another server; I think the disk access is back to normal speed now.
Max and I started upgrading megatron to Ubuntu 12.NN today. We were having some troubles with getting latest python code to run to support the Summary pages stuff.
Its also a nice test to see what CDS tools fail on there, before we upgrade the workstations to Ubuntu 12.
Since its Linux, none of the usual upgrading commands worked, but after an hour or so of reading forums we were able to delete some packages and all the 3rd party packages and get the upgrade to go ahead. We'll have to re-install the LSC, GDS, LAL repos to get it back into shape and get NDS2 working. The upgrade is running in a 'screen' command on there.
Wed Oct 02 14:50:16 2013
Update #1: The upgrade asks a couple dozen questions so it doesn't proceed by itself. I've been checking in to the 'screen' every couple hours to type in 'Yes' to let it keep going.
Update #2: It finished a few hours ago:
controls@megatron:~ 0$ uname -a
Linux megatron 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
controls@megatron:~ 0$ date
Wed Oct 2 18:33:41 PDT 2013
Message on 'pianosa':
Failed to fetch http://ppa.launchpad.net/drgraefy/nds2-client/ubuntu/dists/lucid/main/binary-amd64/Packages.gz 404 Not Found
That's good, but please no more Oplev work. We want to do all of it at once and to make no more changes until we have all the parts (e.g. dumps and correct lenses) in hand and then talk over what the new design will be. I don't want to tune the beam size and loop shape every week.
MC unlocked over the weekend and also got severely mis-aligned. It all started around midnight on Saturday.
At first I thought that this was due to the MCS CPU meter being railed at 60 us, so I deleted a bunch of filters in MC1,2,3 that are unused and left over from Den's quantization noise investigations. This reduced the CPU load somewhat, but didn't make any real improvements. Turning on the ASC filter banks in the MC SUS still mis-aligned the MC.
With the MC WFS and MC ASS turned off, there is still some digital junk coming in and misaligning things. Plot attached.
Similar stuff coming in on ITMX, but not ITMY.
Tried restarting various FEs, but there was no effect. Also tried rebooting c1lsc, c1ioo, & c1sus. Finally did 'shutdown -r now' on all 5 computers on the CDS overview screen and simultaneously (almost) pressed the reset button on the RFM switch above the old c1pem crate. Everything came back OK except for c1oaf (I had to manually button his BURT button) and now the ASC inputs on all the SUS are zero when they should be and MC is well locked and aligned.
Rob and I used to do this trick when he thought that a cosmic ray had corrupted a bit in the RFM network.
I centered the ETMY OL today and found that the UGF was around 3-4x too LOW after the laser swap and re-alignment. That's why the Y arm has been shaking so much today.
NO more OL work without loop measurements and noise measurements.
MC3 watchdog gets tripped sometimes when lock is lost. I noticed that there were no limits set in the MC WFS drive. The attached plot shows that over 40 days, the OUT16 channels from the WFS don't exceed 1000 counts. So I've set the limit to be 2000 in all 6 of the MC ASCPIT/YAW filter banks. Please don't turn them off.
OUT16 is really not the right way to measure this, but for some reason, we don't have any DQ channels from the MC WFS screen ??? So we're not able to measure the trend of the high frequency drive signal.
So I added the WFS(1,2)_I_(PIT,YAW)_OUT_DQ and WFS(1,2)_(PIT,YAW)_OUT_DQ channels to the c1ioo.mdl at 2048 Hz. I used Jamie's excellent 'rtcds' utility to build and install:
1) after making the edits to c1ioo.mdl I saved the file/
2) sshing to c1ioo
3) rtcds stop c1ioo
4) rtcds make c1ioo
5) rtcds install c1ioo
6) rtcds start c1ioo
7) telnet fb 8087
8) daqd> shutdown
That seemed to do it OK.
Unfortunately, all of the instructions that we have in the Wiki for adding channels and model building are misleading and don't mention any of this. There are a few different methods listed which all instruct us to do the whole make and make install business in a bunch of non existent directories.
I noticed that the MC3 LL sensor was apparently dead according to its suspension screen. Since it was only the fast ADC channel and not the SLOW PDmon, I could tell that it was just in the ADC cabling. I pushed in a few of the MC3 sensor cables on the front and back of the PD whitening board and it came back OK. According to this trend of the past 40 days and 40 nights, it started slipping on this past Wednesday morning.
Was anyone walking near MC2 or the suspension electronics racks before noon on Wednesday (Oct. 2nd)?
Tried a bunch of stuff, but eventually just turned off the TRANS_QPD loops and loops are stable. Needs more debugging.
Back around June 18, Jamie was debugging some Guardian code here to replace our MC autolocker. Afterwards our MC WFS stopped working. We never figured out what went wrong, but at the time we turned off the feedback from the MC trans QPD and it stabilized the response at DC.
Today, I noticed that the trans QPD feedback is on. Did anyone do this on purpose?
Its problem causing behavior is slow, but you can catch it if you wait. With the nominal WFS gain of 0.4 the control signal ramps up monotonically at a rate of ~100 counts/minute. Depending upon the static alignment of the MC, this could let it take 10 minutes or a few hours before it rails the MC SUS actuators and breaks the lock. Very sneaky. Don't turn this loop back on without making sure its working and not breaking. I would trend it for you, but the SLOW channels associated with the TRANS QPD servo are not trended --- does anyone know how to get them in the channel list?
I think that we always drive above the UGF for sensing matrix measurements since we like to put notches in the servo. In principle, we could drive within the control band and then take out the effect by measuring the control signal and undoing the gain in the digital filters. But that seems pretty complicated for any MIMO system.