Today I noticed that there was a lot of noise at the Bounce and Roll eigenfrequencies for ETMY. I found that the bandstop filter were set at completely the wrong frequencies, so I've remade them.
The filters were last tuned by Leo in May of 2011. Even so, he left the frequencies at the frequencies of the old MOS suspensions which had f_bounce ~ 12 Hz.
The FOTON plot shows the OLD ones versus the NEW ones. The DTT spectra shows the oplev error signals in the usual state. I have also copied these over to the SUSPOS,PIT,YAW, and SIDE filter banks and turned them all ON.
I also turned OFF and deleted the 3 Hz RG filter that was there. There's no such peak in the error signal and even if one wanted to compensate for the stack mode, it should be a low Q filter, not this monster.
controls@rosalba:/opt/rtcds/caltech/c1/scripts/SUS 0$ ./setOLtramps
Old : C1:SUS-ETMX_OLPIT_TRAMP 0
New : C1:SUS-ETMX_OLPIT_TRAMP 2
Old : C1:SUS-ETMX_OLYAW_TRAMP 0
New : C1:SUS-ETMX_OLYAW_TRAMP 2
Old : C1:SUS-ETMY_OLPIT_TRAMP 2
New : C1:SUS-ETMY_OLPIT_TRAMP 2
Old : C1:SUS-ETMY_OLYAW_TRAMP 2
New : C1:SUS-ETMY_OLYAW_TRAMP 2
Old : C1:SUS-ITMX_OLPIT_TRAMP 0
New : C1:SUS-ITMX_OLPIT_TRAMP 2
Old : C1:SUS-ITMX_OLYAW_TRAMP 0
New : C1:SUS-ITMX_OLYAW_TRAMP 2
Old : C1:SUS-ITMY_OLPIT_TRAMP 0
New : C1:SUS-ITMY_OLPIT_TRAMP 2
Old : C1:SUS-ITMY_OLYAW_TRAMP 0
New : C1:SUS-ITMY_OLYAW_TRAMP 2
Old : C1:SUS-BS_OLPIT_TRAMP 0
New : C1:SUS-BS_OLPIT_TRAMP 2
Old : C1:SUS-BS_OLYAW_TRAMP 0
New : C1:SUS-BS_OLYAW_TRAMP 2
Old : C1:SUS-PRM_OLPIT_TRAMP 0
New : C1:SUS-PRM_OLPIT_TRAMP 2
Old : C1:SUS-PRM_OLYAW_TRAMP 0
New : C1:SUS-PRM_OLYAW_TRAMP 2
Old : C1:SUS-SRM_OLPIT_TRAMP 0
New : C1:SUS-SRM_OLPIT_TRAMP 2
Old : C1:SUS-SRM_OLYAW_TRAMP 0
New : C1:SUS-SRM_OLYAW_TRAMP 2
Done setting TRAMPs
The ETMX oplev signal looks kind of dead compared to the ETMY. It has no features in the spectra and the SUM is pretty low.
I noticed that the cal fields are still set to 1. To get it close to something reasonable, I calibrated it vs. the SUSPIT and SUSYAW values by giving it a step in angle and using 'tdsavg' plus some arithmetic.
OLPIT = 45 urads/ count
OLYAW = 85 urads / count
These are very rough. I don't even know what the accuracy is on the OSEM based calibration, so this ought to be redone in the way that Jenne and Gabriele did before.
The attached image shows the situation after "calibration" of ETMX. This OL system needs some noise investigation.
I centered optical levers of ITMX,BS,ETMY. I also change the position of optical levers of ITMX, ETMY, ITMY, BS on Friday night(9/21), of ITMX, ETMY, BS on Monday night. Both are around 6:00 ~ 7:00.But centering on Monday was totally wrong, because I centered with not good IFO alignment.
The attachment is the 5 days trend of the opt lev of ITMX. First gap is alignment on Friday and Second gap is the alignment on Monday. Now I centered after locking the FPMI.
The attachment 2 is the last 6 hours data. The gap on 9/25 00:00 and 1:30(UTC) is because the alignment of the cavity and the last gap is because of centering of the optical lever.
I went down to investigate the issue with the extra noise that I found in the ETMY optical lever yesterday. There were several problems with the optical layout down there - I'm not sure if I remember them all now.
The main noise issue, however, appears to be not a layout issue at all. Instead its that the laser intensity noise has gone through the roof. See attached spectra of the quadrants (this is the way to diagnose this issue).
I'll ask Steve to either heal this laser or swap it out tomorrow. After that's resolved we'll need another round of layout fixing. I've done a couple of hours today, but if we want a less useless and noisy servo we'll have to do better.
NOTE: by looking at the OL quadrants, I've found a noisy laser, but this still doesn't explain the excess noise in the ETMX. That was the one that has a noisier error signal, not ETMY. By the coherence in the DTT, you can see that the ETMY OL is correctly subtracting and normalizing out the intensity noise of the laser. Seems like the ETMX electronics might be the culprit down there.
We are out of JDSU-Uniphase 1103P heads. I'm ordering some right now. I'm planning to make some corrections on Rana's list tomorrow morning at ETMY.
Not so fast! We need to plan ahead of time so that we don't have to repeat this ETMY layout another dozen times. Please don't make any changes yet to the OL layout.
Its not enough to change the optics if we don't retune the loop. Please do buy a couple of JDSU (and then we need to measure their intensity noise as you did before) and the 633 nm optics for the mode matching and then we can plan about the layout.
As another proof that sometime is ill with ETMX Optical Lever:
We scanned the ETMX bias in PIT using ezcastep and saw that the OL response is very screwy. In the attached, you can see that the ETMX SUSPIT signal shows that the actual motion is good and linear. In fact, our sus diagonalization is extermely good and there's almost no signal in SUSYAW.
ETMY oplev laser clearly showing a tail when it was projected up the sealing.
PS (10-4-2013): I checked the beam quality again as it was removed from the table: it had a good image at 3 meters
We replaced the laser for optical lever of ETMY. And also we aligned the path so that beam spot hits the center for each optics. I attached the spectrum of the SUS-ETMY_OPLEV_SUM, the red curve is with old laser, and blue curve is with the new laser. We also measured without laser so as to measure the QPD dark noise (green curve). The change is significant, and seems much closer to other oplev spectrum.(Brown curve is the oplev spectrum of ITMY)
The new laser is:
Manufacture name: JDSU, Model number: 1103P, Serial number: PA892324
The injection power is 3.7 mW and the out coming power is 197 uW (measured just before the QPD). The output value of the SUS-ETMY_OPLEV_SUM is about 8500.
First we measure 2 spectrum ( including the dark noise). After that we replace the laser and align the optical lever path. We changed the post of one of the mirror (just before the QPD) because the hight was too low. Inside of the chamber is darker than before - actually we had scattering light inside the chamber before. We dumped the reflected light from QPD. And then we measured the spectrum of the oplev output. I also aligned oplev of ETMY after restoring the YARM configuration using IFO configure screen.
We don't know actually what was the problem, laser quality or the scattering light or some clipping. But the oplev seems to be better.
Steve: Atm2 shows increased gains correction later for UGF elog 9206
That's good, but please no more Oplev work. We want to do all of it at once and to make no more changes until we have all the parts (e.g. dumps and correct lenses) in hand and then talk over what the new design will be. I don't want to tune the beam size and loop shape every week.
I centered the ETMY OL today and found that the UGF was around 3-4x too LOW after the laser swap and re-alignment. That's why the Y arm has been shaking so much today.
NO more OL work without loop measurements and noise measurements.
RA: I'm not sure how to interperet this; I think that the SUM channel is divided by the SUM so that this is supposed to be RIN, but not sure. Can someone please take a look into the SUS model and then explain in the elog what the SUM normalization algorithm is?
PRM is dark. PRM and SRM oplev servos are off. ETMY is not centered.
Power spetrum of ETMX and ETMY in the view of oplev pitch and yaw.
For my work designing a cost function, so that I can try out new feedback servo designs on the oplevs, I wanted to know what the dark noise of an oplev is. Since the pitch and yaw channels are divided by the sum channel, when the laser is off, the noise in the pitch and yaw channels looks much higher than it really is. So, I collected some data from the 4 individual quadrants of the ITMY oplev, when the laser was on (but damping was off), and when the laser was off. I used the values of the oplev input matrix to re-create the non-normalized pitch and yaw signals. What I see is that we have some kind of real signal below 1 kHz, but we're hitting the noise at around 1 kHz. So, we definitely don't want to use oplev error signal information above 1 kHz when designing new servos.
The last word in the title is "off". OSEM damping was on, but the oplev damping was off. These are uncalibrated, because the calibrations that we have to go from counts to microradians are for the normalized signals.
I centered the oplev of the ETMY, because I found that Yarm lost lock every 10 minutes, and the ETMY oplev was misaligned very much. I attached the 40 minutes trend of oplev and LSC-TRY. Yarm looks more stable. After centering of the oplev, the YARM looks to be more stable.
Atm1 bandwidth 0.01 Hz ETMY plot is different from
Atm2 bandwidth 0.1 Hz normal ??????? They should be the same !!!!
ETMX _OPLEV_...ERROR signals are effected by turning ON the flow bench motor at the south end
The broadband noise [~ 5-60 Hz] is much higher when the motor is running.
The flow bench was turned off.
Looking into control signals and error signals of the Y arm green PDH servo,
1. The saturation of feedback signal (PZT_OUT) at +/-4000 counts (less than 5V) comes from only the readout saturating. The signals looked fine on the oscilloscope.
2. We did a sine sweep at the PZT_OUT and optimized the LO frequency. The LO frequency did not need any change.
3. The error signal has some offset to it. We are not sure where this comes from.
We have been seeing that whenever green loses lock, the spot position moves down in pitch on the ETMYF camera and the GTRY camera. This led us to think about if the lock loss originated from the PDH or from the cavity.
We looked into dataviewer channels of green, IR, oplev and suspension for the following cases:
1. Green and IR PDH locked
2. Green locked and arms flashing for IR
3. Green shutter closed and IR PDH locked
4. Green shutter closed and arms flashing for IR
5. Arms flashing for IR and ETMY oplev servo turned off.
Dataviewer snapshots of glitch in all the above cases are saved in masayuki's folder users/masayuki/ALS/kicked_mirror/
In all the above cases, we could still see the glitch. We could conclude that the problem lied with the ETMY SUS.
Shown below is the dataviewer snapshot of ETMY sus and shadow sensor channels. The glitch exists even when the oplev servo is turned off pointing to problems associated with the ETMY suspension.
ETMY sensors are glitching or getting kicked up.
Atm3, There is no seismic activity here.
This is not really definitive. The 0.1-0.3 Hz band is not the right one to look for seismic transients - it should be the higher frequency ones.
The other test to do is to turn off the ETMY damping and then look for glitching in the sensors. And then, of course, check to see that no one has bumped the satellite box with a cart or a mop...
I noticed by eye that during one event when ETMY was getting kicked up, its CPU meter (C1:FEC-47_CPU_METER) went RED.
Thinking that this might be a clue I tried to trend this channel. Even though this channel is in the SCY EDCU file and the 'rtcds install' command claims to be 'installing C1EDCU_SCY', many of the channels named in the file are not actually showing up in ur dataviewer SLOW channels list.
I smell a cockroach in our RCG build process, but I can't find the log file for the make-install part of the build nor can I find the Makefile from which the make-install is born. Help us Jamie!
I have deleted a few filters from c1scy to see if that could reduce the CPU time and I have killed the c1tst process to see if that can cool down the entire computer. Next, we can try to open the rack doors and put a fan on there to see if we can shave a couple microseconds. I have a StripTool running on pianosa to see if we see some correlations between FEC47 and the ETMY SUS watchdog RMSs. Don't close it.
c1scy has been running slow (compared to c1scx, which does basically the exact same thing *) for many moons now. We've looked at it but never been able to identify a reason why it should run slower. I suspect there may be some bios setting that's problematic.
The RCG build process is totally convoluted, and really bad at reporting errors. In fact, you need to be careful because the errors it does print are frequently totally misleading. You have to look at the error logs for the full story. The rtcds utility is ultimately just executing the "standard" build instructions. The build directory is:
The build/error logs are:
I'll add a command to rtcds to view the last logs.
(*) the phrase "basically the exact same thing" is LIGO code for "empirically not at all the same"
While that would be good - it doesn't address the EDCU problem at hand. After some verbal emailing, Jamie and I find that the master file in target/fb/ actually doesn't point to any of the EDCU files created by any of the FE machines. It is only using the C0EDCU.ini as well as the *_SLOW.ini files that were last edited in 2011 !!!
So....we have not been adding SLOW channels via the RCG build process for a couple years. Tomorrow morning, Jamie will edit the master file and fix this unless I get to it tonight. There a bunch of old .ini files in the daq/ dir that can be deleted too.
We have now watched the ETMY computer situation for a little over 150 minutes, and have seen one 'event' where the CPU time of the scy model hit 62 microseconds, and a glitch in the ETMY OSEM sensors happened at the same time. We also see no such glitches at any other time, which makes sense with our latest hypothesis, since this event was the only time that the CPU time reported being greater than 61 microseconds. (1/16384 Hz = 61.1696 microseconds).
I have now restarted the c1tst model, to see if that increases the rate of glitches (assuming that running another model heats up the whole computer a bit more, and that makes things run a little bit slower).
Wed Oct 23 21:05:28 2013
RXA: It looks like there was a real effect. Its between -2.5 and 0 on the plot below.
I've stopped the process of c1tst again to make it get better. At 9:20, I also went and opened the front rack door (the back one was already open). One reason its hot may be that the exhaust vents on the top of c1iscey are blocked by one of the custom multi-pin adaptor boxes. In the morning, we should drop the computer down by 1 or 2 notches in the rack so that it can air cool itself better. Make sure to poweroff the computer from the terminal before moving it though.
I checked the cabling somewhat. The fat grey cable which comes out of the old Sander Liu AA chassis was connected to the blue adaptor box but the strain relief screws were not being used. I tightened them (we need to buy a set of small screwdrivers for the toolboxes at each end). While doing this, the Cat6 cable in the back labeled 'c1iscey' popped out and the screen went white. This cable has a broken latch on it so it doesn't stay put - needs to be replaced too during the computer move.
Atm1, The strong glitches are back.
Atm2, SUS-ETMX & Y_SENSOR_LL Damping OFF at (ref0 & ref1). damping ON (red & blue)
ETMY sus looks OK
When PRMI + 2arms are locked yesterday, we heard the noise from suspension violin mode. For attenuation of that noise, we should design the resonant filter at that frequency and put into the ALS servo. I tried to measure the violin mode of ETMs SUS.
What I did
1.The arms were locked by IR PDH. I used awggui to excite the suspention. I injected the Normal waveform, 10 Hz of bandwidth wave into C1:SUS-ETMs_ULCOIL_EXC. I put cheby filter in the FIlter of awggui. The order of that filter was 4, that has same bandwidth as that of injection wave and ripple was 4dB. I increase the injection gain with some ramp time(5sec). I swept from 600 Hz to 700 Hz. During that injection I saw the PDH error signal (POX11I and POY11I) in order to find resonance peak of violin mode.
In ETMX resonances were easily found. That were at 631 Hz and 691 Hz. the 631 Hz peak was seen ALS error signal yesterday. On the other hand, I couldn't find ETMY violin mode. No peaks appeared any frequency.
2. For find the ETMY violin mode, I used dtt swept sine measurement. The excitation channel was C1:SUS-ETMs_ULCOIL_EXC. I measured the TF from excitation channel to POX11I and POY11I error signal. The measurement range was above 400 Hz and below 1000Hz,. The number of point is 600. I attached that result.
In ETMX curve, the coherence become bad near the resonant frequency of violin mode and also the TF is large. Although ETMX violin modes are obvious, ETMY violin modes are not visible. At 660 Hz, 780 Hz, 900 Hz the coherence is not good. That is because 60 Hz comb noise.
I attached the spectrum of the POX and POY error signal. Black and red curve is measured different time. I didn't inject any signal in both measurement, but the violin mode excitation has huge difference. Also there are peaks at beat frequency between violin mode and bounce mode(16 Hz), yaw motion(3 Hz). In ALS in-loop noise or XARM in-loop measurement, sometimes this region had big spikes. That was because of this resonance. And also that resonance peak couples to POY11I.
I will measure the Q and design the resonant filter for ALS.
At some point tonight we lost our CA connection to c1auxex (which is actually the computer at the X End and controls the ETMX, but has a Y sticker). We could telnet to it, but its puny RAM must have been overloaded with too many EPICS connections that bypassed the CArepeater. I went around and booted some machines and it seems to be back and allowing damping now. Along the way I keyed off the crate to c1auxex a couple of times.
When trying to close the rack door I saw that Charlie/Steve had illegally connected the power cable for the illuminator through the door so that it couldn't close, so I disconnected it so that they can run it properly and feel better about themselves.
Disclaimer: Steve had nothing to do with this connection. I rerouted the cable the correct way. 10-28-2013
** what does this coherence tell us about the noise in the arms ?
I thought it would be enough to notch the fundamental and the first harmonic, but sometime tonight the 2nd harmonic at 1892.88 Hz also got rung up.
I made a "Violin3" stopband filter for it and measured its Q using the ole DTT heterodyne secret handshake. Seems much too high to me - it would be nice if someone else would look at this plot and estimate the Q from it.
Turned the PSL HEPA switch back ON - I think its been off for at least a week. I turned the HEPA's variac to 20 after finishing the alignment on the table.
We lowered the c1iscey machine to make space upside of the computer for heat flow.
First we turned off the computer. And then we droped the computer down by 1 notches in the rack. Now the upside and downside spaces are almost same. We restarted the computer after that and we leave the door open.
After some torture Masayuki admitted that he and Steve ignored this elog and just turned off the power button. He blames Steve entirely.
to keep from damaging our computers and our data, NEVER DO THAT.
I used the same OSEM SUSPIT/YAW method as before to calibrate the ETMY optical lever signals. They were off by a factor of ~10.
ETMY Pitch 300 / 26 (old/new) urad/counts
ETMY Yaw 300 / 31 (old/new) urad/counts
These should be redone with the Kakeru / Ottaway arm cavity power technique if we want to get better than ~30% accuracy.
Some more words about the ISS -> OSEM measurement:
The calibration of the OSEMs have been done so that these channels are each in units of microns. The SIDE channel has the lower noise floor because Valera increased the analog gain by 5x some time ago and compensated with lower digital gain.
The peak heights in the plot are:
So that tells us that the coupling is not uniform, but mostly coming in from the left side (which side is the the SIDE OSEM on?).
Jenne and I discussed what to do to mitigate this in the loops. Before we vent to fix the scattering (by putting some covers around the OSEMs perhaps), we want to try to tailor the OSEM damping loops to reduce their strength and increase the strength of the OL loops at the frequencies where we saw the bulk of the instability last time.
Jenne is optimizing OL loops now, and I'm working on OSEM tweaking. My aim is to lower the overall loop gains by ~3-5x and compensate that by putting in some low Q, resonant gain at the pendulum modes as we did for eLIGO. We did it here at the 40m several years ago, but had some troubles due to some resulting instability in the MC WFS loops.
In parallel, Steve is brainstorming some OSEM shields and I am asking around LIGO for some AC OSEM Satellite modules.
IR off for 11 minutes. The PRM face sensors are effected. The PRM side and the rest of the SUS OSEMS are not effected.
Sorry to say but MC1, MC2, MC3 and PRM face OSEMS are having the same problem of leaking IR into the sensors
The PMC was not locked for 11 minutes on this plot.
ETMY sus damping restored
PRM is aligned. IFO is not locked. It is just flashing, including arms. Olympus SP570UZ camera used without IR blocker. Note: PRM side OSEM does not show IR effect.
I will take more pictures with IOO IR blocked and HeNe oplev blocked tomorrow morning.
It crossed my mind that, from these pictures, it could be glow from the oplev scattered light that is causing the problem. However, that seems not possible, since the power fluctuations that we see depend on the presence of the IR light - if it were the oplev light, then when I close the PSL shutter, I should see the same amount of kick, which I don't. Also, the amount of fluctuation increases with increased stored power in the cavities. Also, also, Steve reminds me that some of the MC mirrors see similar kicks in their OSEM signals, but they don't have oplevs.
So, I don't believe that the oplev light is causing the problem, but I wanted to write down why I don't think that's it.
Investigations into OSEM and oplev loops to get rid of the kicks are continuing.
Nice camera work Steve! I will use these for publicity photos.
Now we need to get one of the video cameras hooked into the MUX so that we can see the flashing and do some image subtraction.
Can't we somehow hook up this camera to the MUX with the movie mode?
I think both the MUX and the sensoray are compatible with the color video signal.
Only the old CRT is B/W.
Can't we somehow hook up this camera to the MUX with the movie mode?
I think both the MUX and the sensoray are compatible with the color video signal.
Only the old CRT is B/W.
Watek 902H ccd with Tamron M118FM50 lens is hooked up to MUX Please be careful! In this set up the lens is close to the view port glass window!
Green welding glass is used in these Koji designed dumps (D1102375)
We have 10 pieces of hexagonal dumps for 5.5" high beam They require 1 5/8" space. Atm1
Atm2, Large V traps are 3" tall only, 5 pieces
Atm3, Diamond shapes come with 2" and 1" square green glass ( after Koji's correction I removed the not needed glass ) D1102445 and D1102442
Baked green glass pieces in stock: 30 pieces of 2" x 2" ,--- 30 pieces of 1" x 1",David 4-17-2014
Baked diamond holders in stock: 10 pieces of 2" and 10 pieces of 1"David 4-17-2014
PEEK shims 2" and 1"
Baked green glass pieces blank: 4 pieces of 7" x 9"
Baked green glass pieces with 40 mm hole on 7" x 9" for SUS tower: 7 pieces.
NOTE: in December 2012 we talked about 50 mm aperture need. What diameter is the right one today? 51 mm aperture plates are cut 4-10-2014
You don't need the fourth glass piece on the diamond beam dump.
The PRM sensors are no longer effected by IR. What changed? The MC still does.
ETMY damping restored.
Steve fixed the PRM oplev pointing. I turned on the loops and measured the OLG, then set the pitch and yaw gains such that the upper UGF was ~8Hz (motivated by Jenne's loop design in ELOG 9401)
I then measured the oplev spectra of the optics as they were aligned for PRMI. (OSEMs on, oplevs on, LSC off, and ASC off)
Next, Jenne and I need to fix the ASC loop such that it properly accounts for the oplev loop.
The input pointing of PRM oplev beam was streered just a touch to remove clipping from it's return.
The spots did not move visibly on these two lenses. The spot diameter on the qpd is ~1.5 mm, 65 micro W and 3440 counts.