Hidden in Nakano-kun's previous entries was that the phase margin of the X-Arm was only 9 degrees!! This extremely close to instability and makes for huge gain peaking. The feedback loop is increasing noise above 100 Hz rather than suppress. After some tweaks of the LSC filters we got a much more stable loop/.
So we today started to examine the sources of phase lag in the arm cavity sweeps. There were a few unfortunate choices in the XARM LSC filter bank which we tuned to get less delay.
Then I wrote a bunch of detail about how that worked, but the ELOG ate my entry because it couldn't handle converting my error signal noise plot into a thumbnail. Then it crashed and I restarted it. We also have now propagated the changes to the Y arm by copy/paste the filters and the result there is pretty much the same: low phase margin is now 38 deg phase margin. Noise is less bad.
I fixed the XARM and YARM real time calibration servo.
I also change the C1CAL_MICH_A servo. Now the actuator response and the suspension TF are combined together and that filter name is BS_act. C1CAL_XARM_A and C1CAL_YARM_A have same kind of filters, ETMX_act and ETMY_act.
There are AI filter in each A servo and inv_AA, inv_DAA filters in CINV servo, but it's doesn't work correctly yet.
These aren't servos. What he means is that he's changed some filters in the real time calibration screens so as to make the actuation and sensing parts more accurate, but the inversion of the AA filters is not accurate yet.
I think we can use the IMC autolocker to start with getting this started. Once Jamie fixes the NDSSERVER environment variable bug, we should be able to use his more slick automation code to make it auto lock.
Probably more important is to establish quantitatively how particle counts affects the lock acquisition or noise in the interferometer.
We don't want to adopt a "Sky is Falling" mentality as was done previously in LIGO when people were trying to outlaw burritos and perfume.
Dust monitor counts and human noses do not correlate well with the interferometer's nose.
When the vacuum system is closed, we might check to see if particle counts correlate with losses in the PMC or excess scatter on the ISC tables. If not, we should move on to other concerns.
Yesterday and this morning's slow NFS disk access was caused by 'svndumpfilter' being run at linux1 to carve out the Noise Budget directory. It is being moved to another server; I think the disk access is back to normal speed now.
Max and I started upgrading megatron to Ubuntu 12.NN today. We were having some troubles with getting latest python code to run to support the Summary pages stuff.
Its also a nice test to see what CDS tools fail on there, before we upgrade the workstations to Ubuntu 12.
Since its Linux, none of the usual upgrading commands worked, but after an hour or so of reading forums we were able to delete some packages and all the 3rd party packages and get the upgrade to go ahead. We'll have to re-install the LSC, GDS, LAL repos to get it back into shape and get NDS2 working. The upgrade is running in a 'screen' command on there.
Wed Oct 02 14:50:16 2013
Update #1: The upgrade asks a couple dozen questions so it doesn't proceed by itself. I've been checking in to the 'screen' every couple hours to type in 'Yes' to let it keep going.
Update #2: It finished a few hours ago:
controls@megatron:~ 0$ uname -a
Linux megatron 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
controls@megatron:~ 0$ date
Wed Oct 2 18:33:41 PDT 2013
Message on 'pianosa':
Failed to fetch http://ppa.launchpad.net/drgraefy/nds2-client/ubuntu/dists/lucid/main/binary-amd64/Packages.gz 404 Not Found
That's good, but please no more Oplev work. We want to do all of it at once and to make no more changes until we have all the parts (e.g. dumps and correct lenses) in hand and then talk over what the new design will be. I don't want to tune the beam size and loop shape every week.
MC unlocked over the weekend and also got severely mis-aligned. It all started around midnight on Saturday.
At first I thought that this was due to the MCS CPU meter being railed at 60 us, so I deleted a bunch of filters in MC1,2,3 that are unused and left over from Den's quantization noise investigations. This reduced the CPU load somewhat, but didn't make any real improvements. Turning on the ASC filter banks in the MC SUS still mis-aligned the MC.
With the MC WFS and MC ASS turned off, there is still some digital junk coming in and misaligning things. Plot attached.
Similar stuff coming in on ITMX, but not ITMY.
Tried restarting various FEs, but there was no effect. Also tried rebooting c1lsc, c1ioo, & c1sus. Finally did 'shutdown -r now' on all 5 computers on the CDS overview screen and simultaneously (almost) pressed the reset button on the RFM switch above the old c1pem crate. Everything came back OK except for c1oaf (I had to manually button his BURT button) and now the ASC inputs on all the SUS are zero when they should be and MC is well locked and aligned.
Rob and I used to do this trick when he thought that a cosmic ray had corrupted a bit in the RFM network.
I centered the ETMY OL today and found that the UGF was around 3-4x too LOW after the laser swap and re-alignment. That's why the Y arm has been shaking so much today.
NO more OL work without loop measurements and noise measurements.
MC3 watchdog gets tripped sometimes when lock is lost. I noticed that there were no limits set in the MC WFS drive. The attached plot shows that over 40 days, the OUT16 channels from the WFS don't exceed 1000 counts. So I've set the limit to be 2000 in all 6 of the MC ASCPIT/YAW filter banks. Please don't turn them off.
OUT16 is really not the right way to measure this, but for some reason, we don't have any DQ channels from the MC WFS screen ??? So we're not able to measure the trend of the high frequency drive signal.
So I added the WFS(1,2)_I_(PIT,YAW)_OUT_DQ and WFS(1,2)_(PIT,YAW)_OUT_DQ channels to the c1ioo.mdl at 2048 Hz. I used Jamie's excellent 'rtcds' utility to build and install:
1) after making the edits to c1ioo.mdl I saved the file/
2) sshing to c1ioo
3) rtcds stop c1ioo
4) rtcds make c1ioo
5) rtcds install c1ioo
6) rtcds start c1ioo
7) telnet fb 8087
8) daqd> shutdown
That seemed to do it OK.
Unfortunately, all of the instructions that we have in the Wiki for adding channels and model building are misleading and don't mention any of this. There are a few different methods listed which all instruct us to do the whole make and make install business in a bunch of non existent directories.
I noticed that the MC3 LL sensor was apparently dead according to its suspension screen. Since it was only the fast ADC channel and not the SLOW PDmon, I could tell that it was just in the ADC cabling. I pushed in a few of the MC3 sensor cables on the front and back of the PD whitening board and it came back OK. According to this trend of the past 40 days and 40 nights, it started slipping on this past Wednesday morning.
Was anyone walking near MC2 or the suspension electronics racks before noon on Wednesday (Oct. 2nd)?
Tried a bunch of stuff, but eventually just turned off the TRANS_QPD loops and loops are stable. Needs more debugging.
Back around June 18, Jamie was debugging some Guardian code here to replace our MC autolocker. Afterwards our MC WFS stopped working. We never figured out what went wrong, but at the time we turned off the feedback from the MC trans QPD and it stabilized the response at DC.
Today, I noticed that the trans QPD feedback is on. Did anyone do this on purpose?
Its problem causing behavior is slow, but you can catch it if you wait. With the nominal WFS gain of 0.4 the control signal ramps up monotonically at a rate of ~100 counts/minute. Depending upon the static alignment of the MC, this could let it take 10 minutes or a few hours before it rails the MC SUS actuators and breaks the lock. Very sneaky. Don't turn this loop back on without making sure its working and not breaking. I would trend it for you, but the SLOW channels associated with the TRANS QPD servo are not trended --- does anyone know how to get them in the channel list?
I think that we always drive above the UGF for sensing matrix measurements since we like to put notches in the servo. In principle, we could drive within the control band and then take out the effect by measuring the control signal and undoing the gain in the digital filters. But that seems pretty complicated for any MIMO system.
its time to get the CM servo hardware turned back on. We're going to want to switch it on when we're about ~1/50th of the way up the CARM fringe.
A good way to re-commission it is to lock it to the single arm, using a Pomona box filter to move the arm pole down to the coupled cavity pole frequency.
We got a new computer from Xi computer corp. I am currently installing Ubuntu 10.04 LTS on to it to start with and then will move on to 12 if we can figure out a way to test it besides "I guess it should work?"
Rosalba has been removed and put onto the old Jamie desk. Old Jamie desk also has a Mac Mini running on there.
At the meeting tomorrow we need to decide on a new Italian baby girl name for this new machine.
This is not really definitive. The 0.1-0.3 Hz band is not the right one to look for seismic transients - it should be the higher frequency ones.
The other test to do is to turn off the ETMY damping and then look for glitching in the sensors. And then, of course, check to see that no one has bumped the satellite box with a cart or a mop...
I noticed by eye that during one event when ETMY was getting kicked up, its CPU meter (C1:FEC-47_CPU_METER) went RED.
Thinking that this might be a clue I tried to trend this channel. Even though this channel is in the SCY EDCU file and the 'rtcds install' command claims to be 'installing C1EDCU_SCY', many of the channels named in the file are not actually showing up in ur dataviewer SLOW channels list.
I smell a cockroach in our RCG build process, but I can't find the log file for the make-install part of the build nor can I find the Makefile from which the make-install is born. Help us Jamie!
I have deleted a few filters from c1scy to see if that could reduce the CPU time and I have killed the c1tst process to see if that can cool down the entire computer. Next, we can try to open the rack doors and put a fan on there to see if we can shave a couple microseconds. I have a StripTool running on pianosa to see if we see some correlations between FEC47 and the ETMY SUS watchdog RMSs. Don't close it.
While that would be good - it doesn't address the EDCU problem at hand. After some verbal emailing, Jamie and I find that the master file in target/fb/ actually doesn't point to any of the EDCU files created by any of the FE machines. It is only using the C0EDCU.ini as well as the *_SLOW.ini files that were last edited in 2011 !!!
So....we have not been adding SLOW channels via the RCG build process for a couple years. Tomorrow morning, Jamie will edit the master file and fix this unless I get to it tonight. There a bunch of old .ini files in the daq/ dir that can be deleted too.
I've finished setting up the fstab on Chiara and the upgrade to Ubuntu 12 seems to have gone well enough. She's fast:
but I forgot to make sure to order a dual head graphics card for it. So we'll order some dual DVI gaming card that the company recommends. Until then, its only one monitor.
Still, its ready for testing control room tools on. If everything works OK for a couple weeks, we can go to 12 on all the other ones.
Just in case people were confused, although the PRMI + 2 ALS arms were controlled, we weren't able to bring them in to resonance. They were in some unknown off-resonant state.
We can try to calculate the expected recycling gain (ignoring losses in the PRM) following section F.2.1 of my Manifesto:
T_PRM = 5.6%, R_ARMS ~ 98%, G_PRC ~38.
So the full TRX/TRY powers should be G_PRC/T_PRM = 690.
In our stable configuration, we were sitting at TRX/Y powers of ~5-10. Once in awhile we could get a state where the power was saturating the detectors at ~50 and possibly would have gone up to 100, but it was all oscillation at that point. (we've got to find and notch the ETM violin mode frequencies in the ALS feedback servos.
As we move in towards resonance, we have to now consider all of complications of handing off to various error signals and CARM optical spring compensation and RF saturation that have been discussed in Rob's thesis and Lisa's lock acquisition modeling.
Rather than limp along with a broken SLOW channel system, I fixed it so that the EDCU files made during the RCG build actually get used and added to the channel list (and thereby available in DV and trends).
I first started by adding all of the EDCU files. This completely fails; daqd just doesn't start and gives some weird exceptions.
So I removed a bunch of them and it runs OK now with ~15000 channels. Previously we had ~1500 slow channels.
An in-between config tonight had ~58000 channels and was also running fine, but the connection to the FB would time out when using DV after several minutes. Possibly we can fix this by adding some more RAM to the FB (the DAQD process uses up 45% of the CPU and 39% of the 8 GB of RAM).
Another issue in getting this to work was that there were a bunch of channel name conflicts between the old C0EDCU.ini and the sub-system EDCU files that I was trying to add. I went through by hand and deleted all of the duplicates from the old file. The new frame files are 80 MB, the old ones were 66 MB.
I hope that /frames doesn't become full - not sure how that is wiped...
I went to re-align the beam into the PMC just now. I also tapped all the components between the laser and the PMC; nothing seems suspicious or loose.
The only problem was that someone (probably Steve or Valera) had closed down the iris just downstream of the AOM to ~1-2 mm diameter. This is much too tight! Don't leave irises closed down after aligning. An iris is not to b used as a beam dump. Getting it within a factor of 5-10 of the beam size will certainly make extra noise from clipping/scattering. After opening the iris, the reflected beam onto the PMC REFL camera is notably changed.
Not sure if this will have any effect on our worsening transmission drift, but let's see over the weekend.
I took pictures of this clipping as well as the beam position on Steve's new Retro Position Sensor, but I can't find the cable for the Olympus 570UZ. Steve, please buy a couple more USB data cables of this particular kind so that we don't have to hunt so much if one of the cryo (?) people borrows a cable.
Attachment shows PMC power levels before and after alignment. After alignment, you can see spikes from where I was tapping the mounts in the beamline. We ought to replace the U-100 mount ahead of the AOM with a Polanski
EDIT: Cryo team returns cable - receives punishments. Picture added.
I made the Yoichi laptop into a CDS laptop called 'asia' a few months ago. Somehow I mistakenly gave it the IP address of our little Acer laptop which is called 'farfalla'. This makes farfalla's network not work. I put the old Dell Aldabella by the PMC where farfalla was and am now upgrading farfalla from CentOS to Ubuntu 10.04 LTS 32-bit. I have updated the hostable on linux1 to give farfalla the 230 IP address and let 'asia' keep 225.
I thought it would be enough to notch the fundamental and the first harmonic, but sometime tonight the 2nd harmonic at 1892.88 Hz also got rung up.
I made a "Violin3" stopband filter for it and measured its Q using the ole DTT heterodyne secret handshake. Seems much too high to me - it would be nice if someone else would look at this plot and estimate the Q from it.
Turned the PSL HEPA switch back ON - I think its been off for at least a week. I turned the HEPA's variac to 20 after finishing the alignment on the table.
It would be nice if we could use the existing seismometer cable and place a 2-terminal temperature sensor within the stainless-steel can. A device like the AD590/592 can drive current over a long cable run without pickup issues since its a current source. Inside of the seismometer breakout box we should make a circuit to scale the signal to be close to zero at 25 C and have a slope of 1 V/deg. There are example circuits in the application note - we can just make them on a piece of vector board and glue to the inside of the breakout box (where we connect to the regulated power).
I've stopped the process of c1tst again to make it get better. At 9:20, I also went and opened the front rack door (the back one was already open). One reason its hot may be that the exhaust vents on the top of c1iscey are blocked by one of the custom multi-pin adaptor boxes. In the morning, we should drop the computer down by 1 or 2 notches in the rack so that it can air cool itself better. Make sure to poweroff the computer from the terminal before moving it though.
After some torture Masayuki admitted that he and Steve ignored this elog and just turned off the power button. He blames Steve entirely.
to keep from damaging our computers and our data, NEVER DO THAT.
If so, or if not but you care about the signal that passes through these amplifiers, I suggest you remove this temporary power supply and wire the power from the rack power supplies through the fuse blocks and possibly use a voltage regulator.
In 24 hours, that power supply will be disconnected and the wires snipped if they are still there.
I used the same OSEM SUSPIT/YAW method as before to calibrate the ETMY optical lever signals. They were off by a factor of ~10.
ETMY Pitch 300 / 26 (old/new) urad/counts
ETMY Yaw 300 / 31 (old/new) urad/counts
These should be redone with the Kakeru / Ottaway arm cavity power technique if we want to get better than ~30% accuracy.
FE Web view was broken for a long time. It was fixed now.
The problem was that path names were not fixed when we moved the models from the old local place to the SVN structure.
The auto updating script (/cvs/cds/rtcds/caltech/c1/scripts/AutoUpdate/update_webview.cron) is running on Mafalda.
Link to the web view: https://nodus.ligo.caltech.edu:30889/FE/
Seems partially broken again. Not updating for most of the FE. I've commented out the cron lines for this as well as the mostly broken MEDM Snapshots job. I'm in the process of adding them to the megatron cron (since that machine is at least running 64 bit Ubuntu 12, instead of 32-bit CentOS)
Seems to now be working. I made several fixes to the scripts to get it working again:
I installed 'nfs-client' on zita (the StripTool terminal). It now has mounted all the shared disks, but still can't do StripTool since its a 32-bit machine and our StripTool is 64.
The first picture shows that there is indeed a DAC next to the ADC in the LSC IO chassis. The second picture shows how there are two cables, each one carrying 8 channels of DAC. The third one shows how these come out of the coil drivers to handle the Tip/Tilt mirrors which point the beam from the IMC into the PRC. It should be the case that the second Dewhitening filter board can give us access to the next 8 channels for use in driving an audio signal into the control room or an ISS excitation.
Back in 2009, Jenne replaced the PMC board mixer with a Level 13 one. Today I noticed that the LO level on the PMC screen was showing a LO level of ~5-10 dBm and fluctuating a lot. I think that it is related to the well known failure of the Mini-Circuits ERA-5SM amplifier which is on the D000419-A schematic (PMC Frequency Reference Card). The Hanford one was dying for 12 years and we found it in late 2008. If we don't have any in the blue bin, we should ask Steve to order 10 of them.
The attached trend shows 2000 days of hour trend of the PMC LODET channel. The big break in 2009 is when Jenne changed the mixer and then attenuated the input by 3 dB. The slow decay since then is the dying amplifier I guess.
Since the LOCALC channel was not in the trend, I added it to the C0EDCU file tonight and restarted the FB DAQD process. Its now in the dataviewer list.
I went out and took out the 3 dB attenuator between the LO card and the PMC Mixer. The LO monitor now reads 14.9 dBm (??!!). The SRA-3MH mixer data sheet claims that the mixer works fine with an LO between 10 and 16 dBm, so I'll leave it as is. After we get the ERA-5, lets fix the LODET monitor by upping its gain and recalibrating the channel.
Some more words about the ISS -> OSEM measurement:
The calibration of the OSEMs have been done so that these channels are each in units of microns. The SIDE channel has the lower noise floor because Valera increased the analog gain by 5x some time ago and compensated with lower digital gain.
The peak heights in the plot are:
So that tells us that the coupling is not uniform, but mostly coming in from the left side (which side is the the SIDE OSEM on?).
Jenne and I discussed what to do to mitigate this in the loops. Before we vent to fix the scattering (by putting some covers around the OSEMs perhaps), we want to try to tailor the OSEM damping loops to reduce their strength and increase the strength of the OL loops at the frequencies where we saw the bulk of the instability last time.
Jenne is optimizing OL loops now, and I'm working on OSEM tweaking. My aim is to lower the overall loop gains by ~3-5x and compensate that by putting in some low Q, resonant gain at the pendulum modes as we did for eLIGO. We did it here at the 40m several years ago, but had some troubles due to some resulting instability in the MC WFS loops.
In parallel, Steve is brainstorming some OSEM shields and I am asking around LIGO for some AC OSEM Satellite modules.
I used our procedure from this entry to set the IMC board offset as well as the FSS board offset.
I found this afternoon that the MC was having trouble locking: the PC path was railing as soon as the boost was engaged. Could be that there's some misalignment on the PSL which has led to some RAM having to be canceled by this new offset. Let's see if its stable for awhile.
I felt in my bones that the MC was in trouble so I came by and noticed that it hadn't locked for a couple hours. The FSS SLOW was at -1.6V, but putting it back to zero didn't fix things. I adjusted the FSS error point offset to +1 and that took the FSS_FAST off of the +10 V rail. Relocked and seems OK.
We need to plan to make the M Evans mod to the FSS box to make the PC drive less angry.
Last 40 days of MC Alignment trends show that the recent MC WFS tuning / offseting worked out OK. MC REFL seems low and flat.
It seems to me that current design of the common mode servo is already fine. Attached plots show common mode open and closed loop transfer function.
These seem like pretty terrible loop shapes. Can you give us a plot with the breakdown of several of the TFs and some .m file?
We should be able to estimate the noise coming out of the MC using the single arm and then make a guess for the CM loop gain requirement. There's no reason to keep the old Boost shapes; those were used in the old MC configuration which had a RefCav. In addition to minimizing the EOM range, we should also minimize the AO signal as Koji has pointed out. In practice, I've seen that using ~300 Hz of offset makes no harm with 4 kHz MC pole.
From Linda and Bram:
I looked at the BBPD design so that we could make a POP22/110. It looks like it will be easy (I hope).
The first attachment shows the schematic with the RF notch modified to handle 55 MHz. As long as the capacitor in this notch can be kept to below 20 pF, it doesn't degrade the noise so much,
The second attachment shows the TF and input referred noise. We ought to be able to get 20 pA/rHz at the input to the first RF amplifier.
The LISO files are in the svn under liso/examples/aLIGO_BBPD/,
Later, if we have to notch more than just 55 MHz, we can add a notch between the 2 RF amplifiers as Koji has done for the REFL165.
Since we use the TransMon QPD for triggering the high/low gain switching we need to run with the whitening OFF during lock acquisition and the turn it on after we have the arms locked with ALS. This should be put into the up/down scripts.
This is a 10-minute trend of the last 60 days of the pointing of the PSL beam.
The main fluctuation seems to be at the ~30 day time scale (not 24 hour) and its all in the vertical direction; the horizontal drift is ~10x less (as long as we believe there is no calibration error).
So what's causing all of this vertical shift? And why is there not just as much horizontal??
I went to the PSL table to re-align the input pointing to the IMC. After trying to optimize the pointing into the PMC and not succeeding I also then touched the wrong mirror and messed up our IOO QPD reference pointing.
The IMC is locking again, but I'll have to fix the pointing on Monday.
NOT drift. The sudden steps are certainly the result of being kicked. The slow drift at the end of the day might be a slow strain relaxation.
It pays to be careful and not put too much weight or impulsive forces on the chambers or tables.
For the IPANG telescope design, we are in the 'beyond the Rayleigh range' regime. So using a single lens to make the beam small is not a great idea. I
Can you please explain this? I don't understand what exactly is the issue or 'great idea'.
I think we should be OK with just a single lens in the vacuum. But what we need is the ray tracing analysis to show what the effect will be on the IPANG readout.
Its very doubtful that the MC yaw drift matters for the IFO. That's just a qualitative correlation; the numbers don't hang together.
Since the recent filesystem fracas, the new accounts could not be created on nodus / dokuwiki (for the controls workshop, for example).
I started sendmail on nodus using the command: sudo /etc/init.d/sendmail start
sudo /etc/init.d/sendmail start
and the SwiftMail plugin on there is now sending out the confirmation emails again. This will happen each time we reboot nodus, so let's replace it.
The ELOG was frozen, with this in the .log file:
GET /40m/?id=1279&select=1&rsort=Type HTTP/1.1
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
(hopefully there's a way to hide from the Bing Bot like we did from the Google bot)