I had some problem with the Oplev Servo today. I was working at the mode matching fine tuning and I left the Oplev servo enabled while aligning.
When I opened the Yend table lids, the Oplev beam started moving on the QPD and the Oplev servo didn't help in stopping the mirror movement, but it increased it.
So, the mirror was oscillating at a frequency of a few Hz
Koji suggested that maybe the shaking is due to the air conditioning moving the beam, so the servo tries to feed back the signal to the mirror, even if the mirror doesn't actually move.
I also measured the transfer function for the Oplev, but it didn't show any strange behavior.
I've restarted the NDS2 process on Megatron so that we can use it for getting past data and eventually from outside the 40m.
1) from /home/controls/nds2 (which is not a good place for programs to run) I ran nds2-megatron/start-nds2
2) this is just a script that runs the binary from /usr/bin/ and then leaves a log file in ~/nds2/log/
3) I tested with DTT that I could access megatron:31200 and get data that way.
There is a script in usr/bin called nds2_nightly which seems to be the thing we should run by cron to get the channel list to get updated, but I' m not sure. Let's see if we can get an ELOG entry about how this works.
Then we want Jamie to allow some kind of tunneling so that the 40m data can be accessed from outside, etc.
I have done the following:
* installed the nds2-client in /ligo/apps/nds2-client
* moved the nds2 configuration directories to /ligo/apps/nds2/nds2-megatron
* set up a cron job to update the channel list every morning at 5 am. The cron line is:
15 5 * * * /usr/bin/nds2_nightly /ligo/apps/nds2/channel-tracker /ligo/apps/nds2/nds2-megatron
cron will send an email each time the channel list changes, at which point you will have to restart the server with:
* restarted nds2 with updated channel lists.
I have set the cron job up to restart the nds2 server automatically if the channel list changes. The only change is that the cron command was changes to /ligo/apps/nds2/nds2-megatron/test-restart.
Now that the 3f locking looks so cool for the PRMI, I suppose that the PRMI + arm stuff will be very successful.
At LLO, I've just noticed the screens that they have for the single pendulums / TTs. I'm attaching a screenshot of the one Zach is using for the steering into the OMC. We should grab these and replace our existing SUS screens with them.
Totally agree. The old suspension screen should be driven away.
SR2 is flipped, and reinstalled. We did that before lunch, and we're about to go in and work on SR3 and PR3.
EDITS / Notes:
I set dog clamps to have a reference position of where the tip tilt was, then I removed SR3 from the chamber. Once out, I followed the same procedure I used for PR2 during the last vent - I removed the whole suspension (top mount, wires, optic) from the cage, and laid it down flat. Then I loosened the set screw which pushes on the teflon nudge, removed the mirror, inspected it, and put it back in, with the HR side facing the back side of the ring. Then I replaced the suspension system in the cage, and put the mirror back into the chamber.
When I loosened the teflon nudge at the top of the mirror holder ring, the optic seemed to fall down a tiny bit. I think this implies that the HR surface of the optic did not used to be parallel to the front face of the mirror holder ring. When I put the suspension back onto the cage, the pitch balancing was very bad. We checked the level of the table that I had the cage on, and it was miraculously pretty level, so I did the pitch balancing out of the chamber.
Also, during my quick inspection of the mirror (not thorough, just using room lights), I noticed a small fleck of lint near the edge of the optic on the HR surface. The HR surface is now on the outside of the SRC, but we should still blow at the optic with the ionized nitrogen to get it off.
I did not think to check the fine-tuning alignment of SR2....Koji did that after lunch (which I will elog about in a separate elog).
After the first flipping, X/Y arms were aligned and locked. Then the ASS aligned the arms.
I started poking around at what we want for new SUS MEDM screens. Rana and I decided we'd start with the ASC TIPTILT screens:
It's missing some things (like SIDE OSEMS) but it should provide a good starting point.
I copied the entire <userapps>/asc/common/medm/asctt directory to a new directory in our sus area:
controls@rossa:/opt/rtcds/userapps/release 0$ cp -a asc/common/medm/asctt sus/c1/medm/new
I then removed all the useless file name prefixes. We still need to go through and sed out all the ASC stuff in the MEDM files themselves.
It makes heavy use of macro substitution, which is good (it's what we're using now). So once we clean up all the channel names, we should just be able to swap out the pointers in our overview screens to the new screens (or rename things). In the mean time, during development, you can run:
controls@rossa:/opt/rtcds/userapps/release 0$ medm -x -macro "IFO=C1,ifo=c1,OPTIC=ITMX" sus/c1/medm/new/OVERVIEW.adl
Yesterday afternoon, I went back into the BS chamber, and flipped both PR3 and SR3. Now all of the recycling cavity folding mirrors have been flipped.
For PR3, I followed the same procedure as SR2, setting a reference position, removing the optic, flipping it, etc. When I put it back in, I realized that since this has a 41 degree angle of incidence, the beam going to the BS had translated north by ~1cm. After some fiddling, Koji pointed out that the 2 degree wedge probably had a more significant effect than just the HR surface having moved back a small amount. Anyhow, we adjusted PR3 such that we were going through the BS aperture, as well as the ITMY aperture.
During the flip of PR3, Annalisa and I noticed that the arrow on the barrel of the LaserOptik mirrors also indicates the thickest part of the wedge. This is opposite of our SOS optics, where the arrow's position on the barrel indicates the thinnest part of the wedge. For both PR3 and SR3, I kept the arrow on the same side of the optic as it was originally.
I then flipped SR3, following again the same procedure. PR3 I had done a tiny bit of pitch rebalancing, although I think it was unneccessary, since it is within what we can do with the poking/hysterisis method. SR3 I did not do any pitch rebalancing. With PR3 aligned at least to the ITM, Koji and I aligned SR3 and SR2 so that the AS beam was hitting the center of all the SRC optics. We also adjusted the steering mirrors after the SRM to get the beam centered on PZT3, the last optic on the BS table, which launches the beam over to the OMC chamber. We scanned around a bit by turning the PZT's knobs, but we were unable to see the AS beam on the camera.
We have looked a little more at the SRM situation. We aligned the SRM, and then aligned the oplev, so that we had a convenient monitor of the optic's motion.
When we use the _COMM channels, which are the usual ones on the IFO_ALIGN screen, the pitch slider makes pitch motion, but the yaw slider makes the oplev spot move ~45degrees from horizontal.
However, when we use the bias channels that are in the front end model, parallel to the ASC path, pitch moves pitch, and yaw moves pure yaw.
So, we conclude that the SRM coils are fine, and there is something funny going on with the slow part of the actuation.
Koji restarted the slow computer susaux, and burt restored it, but that did not fix the situation. We went inside and looked at all of the ribbon cable connections, and pushed them all in, but that also has not fixed things.
We have been looking at D010001-b, the coil driver board, and we think that's where the summing resistor network between the slow bias slider, and the coil outputs from the fast model exists. (It's not 100% clear, but we're confident that that's what is going on).
Tomorrow, we will pull the SRM's coil driver board, and see if any of the components in the slow slider path, before the summing point, look burned / broken / bad.
Now the SRM Yaw bias in yaw is functional without any strage behavior.
The problem was found at the connector of the flat ribbon cable from the DAC to the cross connect.
I used the extender board to diagnose the SRM coil driver circuit at 1X4.
The UL coil input did not show any sign of voltage no matter how the bias slider was jiggled.
I opened the side panel of the rack and found the signal was absent at the cross connect which relays two flat ribbon cables
for the SRM coil driver. I checked the DAC output with a multimeter. All the bias outputs were OK at the DAC.
Then I opened the IDC connector at the DAC side of the crossconnect as the signal was already missing there.
I found that the flat ribbon cable was a half line shifted from the supposed location.
This resulted a short circuit of the DAQ +/- pins for the SRM UL coil.
I recrimped the connector and now the SRM Yaw slider is back.
This changed the nominal position of the SRM. The new slider values were saved.
ETMX sus damping restored
It seems that the PRM violin mode freqs shifted from 625-ish to 640Hz.
The peaks rang up because of the servo.
Once the notch freq was shifted to 640Hz, the violin mode started to decay.
We were meditating a little bit on what may be the story behind the PRM violin filter situation. We locked the PRMI, and turned on and off the violin filters. We noticed, very bizarrely, that when the violin filters were ON, the servos would oscillate. Weird. Also, probably because the oscillation was causing us to hit the limit we have in the MICH servo, we rung up a 3rd harmonic of one of the violin modes, which was at 1955 Hz.
We took a transfer function of the PRCL servo, saw that the UGF was 300 Hz, and lowered it to ~180 Hz. After later investigations, that high-ish UGF probably wasn't a problem. Anyhow, we then took MICH servo transfer functions, and saw some very weird stuff.
At frequencies where we had violin filter notches, we were seeing peaks in the transfer function, which came close to touching, or crossed the 0dB line! We suspect that this may have something to do with the balancing of the drives to the optics, since we have PRCL driving PRM, but MICH driving BS and PRM. What we did was move the violin filter notches into the LSC model. There were already SUS filter banks in the LSC model (right side of the LSC screen). In preparation for the DRMI, I have put the BS violin notches into the BS, PRM and SRM filter banks, as well as the PRM and SRM filters into all 3 banks. Right now for PRMI, I have the BS and PRM notches (as well as the Vio3 notch) turned on in both BS and PRM. All of the violin-related filters are turned off in the LSC filter bank inside the SUS models. When we did this, the servo oscillations no longer are excited when we turn on the notches, and when we take a new transfer function, there are no longer weird peaks at the notch frequencies. More meditation needs to be done.
Also, for the violin3 filter, Rana noted that at 1955 Hz, after we confirmed that the REFL 55 phase was set correctly (and we're using REFL 55 I&Q for PRMI locking), the I-phase had a response of 0.36, while the Q-phase had a response of 0.20. I should be able to think about these numbers, and decide if the vio3 is for the BS or the PRM violin mode.
the above series of Bode plots shows the MICH Open Loop Gain as the REFL55 phase is adjusted (PURPLE, ORANGE) with the notches in the SUS and then (RED) as the notches are moved to the LSC and made the same for all optics.
In other news, I have the parts that Jamie ordered to make a new 110 demod board, so that's one of my daytime activities now, so we can have both POP110 and AS110.
Just to rephrase somewhat:
PRM and ITMY were found with their watchdogs shutdown this afternoon (cause unknown). I re-engaged them.
Rana asked me to look at the SUS MEDM screen upgrade situation, and provide an upgrade prescription. Unfortunately there not really a simple prescription that can be used, since our configuration diverges quite a bit from what's at the sites. But here's what I can say:
It looks like we already have the beginnings of an upgrade in place, so I say we just run with that. The new screens are in:
The primary screen is:
This seems to be a copy of the site ASC_TIPTILT screens. (In fact I think I remember putting this here). I went ahead and did some ground work to make it easier to get these new screens into place.
At this point someone needs to just go through and fix all the channel names to match ours, and tweak the screen to our needs (there's no side OSEM, for instance). The subscreens need to be cleaned up as well.
If there is a specific string you want to replace every instance of in the screen, you can do that easily from the command line like this:
sed -i 's/OLD/NEW/g'
This will replace every instance of the string OLD with the string new in the file path/to/file. Be careful: this will replace EVERY instance of OLD. If OLD matches things you don't want, they will be replaced as well.
This construction is actually "regular expressions", so if you want to get fancy you can match against more complicated strings. But just be careful.
If you leave out the "-i" the string-replaced text will go to stdout, instead of being replaced in the file "in place", so you can check it first.
If you want more fine-grained control of text replace, so that you can see what's being replaced, try using "query-replace" in emacs:
You can then type in the original string, followed by the replacement string. When it starts to run it will highlight the string that will be replaced. Hit "space" to accept or "n" to skip and go to the next.
While we were trying to relock the MC after Jenne put back the RF box, we found there was some mysterious motion in MC2. After spending time trying to figure out where this was coming from, the source was found to be at LOCKIN2 of MC2 suspension "The MC TICKLER" that was left enabled. This was turned OFF and MC locked just fine after that.
EDIT JCD: The Tickler should be disabled, if the autolocker is disabled.
Sounds like this was just incidental, since the MC locked fine also with the tickler enabled for weeks.
The tickle is disabled by the down script, but there's no way to correctly handle all possible button pushes. If you want to disable the autolocker for some reason you should run mcdown before trying to lock. This will set up things with the correct settings.
Isn't that backwards? Shouldn't the tickler be enabled by the down script, and disabled by the up script? Either way, the problem was that, after I acquired lock, the tickler was causing the transmitted power to fluctuate by ~20%, and often the MC would lose lock after a minute or so. Also, the WFS definitely couldn't be enabled by hand.
Anyhow, I'll try to keep in mind that this exists, so I'm not stymied by it again.
You're right - down turns it on. Still, the fact that the same tickle recently causes a problem and didn't make 20% power fluctuations until now tells me that its not that the tickle amplitude is too large. Whatever changed recently is the problem.
I think the crane repair guy accidentally stepped on the BSC support beam.
PRM side is not coming down.
I think the crane repair guy accidentally stepped on the BS support beam.
Seems fine now.
I'm working on it.
Today I noticed that there was a lot of noise at the Bounce and Roll eigenfrequencies for ETMY. I found that the bandstop filter were set at completely the wrong frequencies, so I've remade them.
The filters were last tuned by Leo in May of 2011. Even so, he left the frequencies at the frequencies of the old MOS suspensions which had f_bounce ~ 12 Hz.
The FOTON plot shows the OLD ones versus the NEW ones. The DTT spectra shows the oplev error signals in the usual state. I have also copied these over to the SUSPOS,PIT,YAW, and SIDE filter banks and turned them all ON.
I also turned OFF and deleted the 3 Hz RG filter that was there. There's no such peak in the error signal and even if one wanted to compensate for the stack mode, it should be a low Q filter, not this monster.
controls@rosalba:/opt/rtcds/caltech/c1/scripts/SUS 0$ ./setOLtramps
Old : C1:SUS-ETMX_OLPIT_TRAMP 0
New : C1:SUS-ETMX_OLPIT_TRAMP 2
Old : C1:SUS-ETMX_OLYAW_TRAMP 0
New : C1:SUS-ETMX_OLYAW_TRAMP 2
Old : C1:SUS-ETMY_OLPIT_TRAMP 2
New : C1:SUS-ETMY_OLPIT_TRAMP 2
Old : C1:SUS-ETMY_OLYAW_TRAMP 2
New : C1:SUS-ETMY_OLYAW_TRAMP 2
Old : C1:SUS-ITMX_OLPIT_TRAMP 0
New : C1:SUS-ITMX_OLPIT_TRAMP 2
Old : C1:SUS-ITMX_OLYAW_TRAMP 0
New : C1:SUS-ITMX_OLYAW_TRAMP 2
Old : C1:SUS-ITMY_OLPIT_TRAMP 0
New : C1:SUS-ITMY_OLPIT_TRAMP 2
Old : C1:SUS-ITMY_OLYAW_TRAMP 0
New : C1:SUS-ITMY_OLYAW_TRAMP 2
Old : C1:SUS-BS_OLPIT_TRAMP 0
New : C1:SUS-BS_OLPIT_TRAMP 2
Old : C1:SUS-BS_OLYAW_TRAMP 0
New : C1:SUS-BS_OLYAW_TRAMP 2
Old : C1:SUS-PRM_OLPIT_TRAMP 0
New : C1:SUS-PRM_OLPIT_TRAMP 2
Old : C1:SUS-PRM_OLYAW_TRAMP 0
New : C1:SUS-PRM_OLYAW_TRAMP 2
Old : C1:SUS-SRM_OLPIT_TRAMP 0
New : C1:SUS-SRM_OLPIT_TRAMP 2
Old : C1:SUS-SRM_OLYAW_TRAMP 0
New : C1:SUS-SRM_OLYAW_TRAMP 2
Done setting TRAMPs
The ETMX oplev signal looks kind of dead compared to the ETMY. It has no features in the spectra and the SUM is pretty low.
I noticed that the cal fields are still set to 1. To get it close to something reasonable, I calibrated it vs. the SUSPIT and SUSYAW values by giving it a step in angle and using 'tdsavg' plus some arithmetic.
OLPIT = 45 urads/ count
OLYAW = 85 urads / count
These are very rough. I don't even know what the accuracy is on the OSEM based calibration, so this ought to be redone in the way that Jenne and Gabriele did before.
The attached image shows the situation after "calibration" of ETMX. This OL system needs some noise investigation.
I centered optical levers of ITMX,BS,ETMY. I also change the position of optical levers of ITMX, ETMY, ITMY, BS on Friday night(9/21), of ITMX, ETMY, BS on Monday night. Both are around 6:00 ~ 7:00.But centering on Monday was totally wrong, because I centered with not good IFO alignment.
The attachment is the 5 days trend of the opt lev of ITMX. First gap is alignment on Friday and Second gap is the alignment on Monday. Now I centered after locking the FPMI.
The attachment 2 is the last 6 hours data. The gap on 9/25 00:00 and 1:30(UTC) is because the alignment of the cavity and the last gap is because of centering of the optical lever.
I went down to investigate the issue with the extra noise that I found in the ETMY optical lever yesterday. There were several problems with the optical layout down there - I'm not sure if I remember them all now.
The main noise issue, however, appears to be not a layout issue at all. Instead its that the laser intensity noise has gone through the roof. See attached spectra of the quadrants (this is the way to diagnose this issue).
I'll ask Steve to either heal this laser or swap it out tomorrow. After that's resolved we'll need another round of layout fixing. I've done a couple of hours today, but if we want a less useless and noisy servo we'll have to do better.
NOTE: by looking at the OL quadrants, I've found a noisy laser, but this still doesn't explain the excess noise in the ETMX. That was the one that has a noisier error signal, not ETMY. By the coherence in the DTT, you can see that the ETMY OL is correctly subtracting and normalizing out the intensity noise of the laser. Seems like the ETMX electronics might be the culprit down there.
We are out of JDSU-Uniphase 1103P heads. I'm ordering some right now. I'm planning to make some corrections on Rana's list tomorrow morning at ETMY.
Not so fast! We need to plan ahead of time so that we don't have to repeat this ETMY layout another dozen times. Please don't make any changes yet to the OL layout.
Its not enough to change the optics if we don't retune the loop. Please do buy a couple of JDSU (and then we need to measure their intensity noise as you did before) and the 633 nm optics for the mode matching and then we can plan about the layout.
As another proof that sometime is ill with ETMX Optical Lever:
We scanned the ETMX bias in PIT using ezcastep and saw that the OL response is very screwy. In the attached, you can see that the ETMX SUSPIT signal shows that the actual motion is good and linear. In fact, our sus diagonalization is extermely good and there's almost no signal in SUSYAW.
ETMY oplev laser clearly showing a tail when it was projected up the sealing.
PS (10-4-2013): I checked the beam quality again as it was removed from the table: it had a good image at 3 meters
We replaced the laser for optical lever of ETMY. And also we aligned the path so that beam spot hits the center for each optics. I attached the spectrum of the SUS-ETMY_OPLEV_SUM, the red curve is with old laser, and blue curve is with the new laser. We also measured without laser so as to measure the QPD dark noise (green curve). The change is significant, and seems much closer to other oplev spectrum.(Brown curve is the oplev spectrum of ITMY)
The new laser is:
Manufacture name: JDSU, Model number: 1103P, Serial number: PA892324
The injection power is 3.7 mW and the out coming power is 197 uW (measured just before the QPD). The output value of the SUS-ETMY_OPLEV_SUM is about 8500.
First we measure 2 spectrum ( including the dark noise). After that we replace the laser and align the optical lever path. We changed the post of one of the mirror (just before the QPD) because the hight was too low. Inside of the chamber is darker than before - actually we had scattering light inside the chamber before. We dumped the reflected light from QPD. And then we measured the spectrum of the oplev output. I also aligned oplev of ETMY after restoring the YARM configuration using IFO configure screen.
We don't know actually what was the problem, laser quality or the scattering light or some clipping. But the oplev seems to be better.
Steve: Atm2 shows increased gains correction later for UGF elog 9206
That's good, but please no more Oplev work. We want to do all of it at once and to make no more changes until we have all the parts (e.g. dumps and correct lenses) in hand and then talk over what the new design will be. I don't want to tune the beam size and loop shape every week.
I centered the ETMY OL today and found that the UGF was around 3-4x too LOW after the laser swap and re-alignment. That's why the Y arm has been shaking so much today.
NO more OL work without loop measurements and noise measurements.
RA: I'm not sure how to interperet this; I think that the SUM channel is divided by the SUM so that this is supposed to be RIN, but not sure. Can someone please take a look into the SUS model and then explain in the elog what the SUM normalization algorithm is?
PRM is dark. PRM and SRM oplev servos are off. ETMY is not centered.
Power spetrum of ETMX and ETMY in the view of oplev pitch and yaw.
For my work designing a cost function, so that I can try out new feedback servo designs on the oplevs, I wanted to know what the dark noise of an oplev is. Since the pitch and yaw channels are divided by the sum channel, when the laser is off, the noise in the pitch and yaw channels looks much higher than it really is. So, I collected some data from the 4 individual quadrants of the ITMY oplev, when the laser was on (but damping was off), and when the laser was off. I used the values of the oplev input matrix to re-create the non-normalized pitch and yaw signals. What I see is that we have some kind of real signal below 1 kHz, but we're hitting the noise at around 1 kHz. So, we definitely don't want to use oplev error signal information above 1 kHz when designing new servos.
The last word in the title is "off". OSEM damping was on, but the oplev damping was off. These are uncalibrated, because the calibrations that we have to go from counts to microradians are for the normalized signals.
I centered the oplev of the ETMY, because I found that Yarm lost lock every 10 minutes, and the ETMY oplev was misaligned very much. I attached the 40 minutes trend of oplev and LSC-TRY. Yarm looks more stable. After centering of the oplev, the YARM looks to be more stable.
Atm1 bandwidth 0.01 Hz ETMY plot is different from
Atm2 bandwidth 0.1 Hz normal ??????? They should be the same !!!!
ETMX _OPLEV_...ERROR signals are effected by turning ON the flow bench motor at the south end
The broadband noise [~ 5-60 Hz] is much higher when the motor is running.
The flow bench was turned off.
Looking into control signals and error signals of the Y arm green PDH servo,
1. The saturation of feedback signal (PZT_OUT) at +/-4000 counts (less than 5V) comes from only the readout saturating. The signals looked fine on the oscilloscope.
2. We did a sine sweep at the PZT_OUT and optimized the LO frequency. The LO frequency did not need any change.
3. The error signal has some offset to it. We are not sure where this comes from.
We have been seeing that whenever green loses lock, the spot position moves down in pitch on the ETMYF camera and the GTRY camera. This led us to think about if the lock loss originated from the PDH or from the cavity.
We looked into dataviewer channels of green, IR, oplev and suspension for the following cases:
1. Green and IR PDH locked
2. Green locked and arms flashing for IR
3. Green shutter closed and IR PDH locked
4. Green shutter closed and arms flashing for IR
5. Arms flashing for IR and ETMY oplev servo turned off.
Dataviewer snapshots of glitch in all the above cases are saved in masayuki's folder users/masayuki/ALS/kicked_mirror/
In all the above cases, we could still see the glitch. We could conclude that the problem lied with the ETMY SUS.
Shown below is the dataviewer snapshot of ETMY sus and shadow sensor channels. The glitch exists even when the oplev servo is turned off pointing to problems associated with the ETMY suspension.
ETMY sensors are glitching or getting kicked up.
Atm3, There is no seismic activity here.
This is not really definitive. The 0.1-0.3 Hz band is not the right one to look for seismic transients - it should be the higher frequency ones.
The other test to do is to turn off the ETMY damping and then look for glitching in the sensors. And then, of course, check to see that no one has bumped the satellite box with a cart or a mop...
I noticed by eye that during one event when ETMY was getting kicked up, its CPU meter (C1:FEC-47_CPU_METER) went RED.
Thinking that this might be a clue I tried to trend this channel. Even though this channel is in the SCY EDCU file and the 'rtcds install' command claims to be 'installing C1EDCU_SCY', many of the channels named in the file are not actually showing up in ur dataviewer SLOW channels list.
I smell a cockroach in our RCG build process, but I can't find the log file for the make-install part of the build nor can I find the Makefile from which the make-install is born. Help us Jamie!
I have deleted a few filters from c1scy to see if that could reduce the CPU time and I have killed the c1tst process to see if that can cool down the entire computer. Next, we can try to open the rack doors and put a fan on there to see if we can shave a couple microseconds. I have a StripTool running on pianosa to see if we see some correlations between FEC47 and the ETMY SUS watchdog RMSs. Don't close it.