In the morning, Steve will start opening the north BS door so that we can enter to inspect the PRM LR OSEM.
For the ITMY, I squished together the cables which are in the 'Cable Interface Board' which lives in the rack. This thing takes the 64 pin IDC from the satellite module and converts it into 2 D-sub connectors to go to the PD whitening board and the coil driver board. Lets see if the ITMY OSEM glitches change character overnight.
[Teng, Johannes, Lydia, gautam]
The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
We peeked into the BS-PRM chamber via the ITMX chamber to see if we could shed any light on this situation. It's hard to get a picture that is in focus, but it looks quite clear that the LR LED (in the lower left when viewed from the HR side) isn't anywhere near as bright as the rest (see Attachment #1). Various hypothesis include failed LED / piece of Al foil blocking the LED / teflon aperture slipped over the LED. But looks like we can't solve this without opening up the BS-PRM chamber. The plan tomorrow is to open up the chamber, pull out the problematic coil. Once we have a better idea of what is going wrong, we can decide what the appropriate course of action is - replace the OSEM or something else.
As part of the diagnosis, I switched the PRM and SRM satellite boxes earlier today evening around 6pm. They remain in this switched state for now.
Steve, we plan to take the BS-PRM heavy door off tomorrow morning.
Edit 7.30pm: I have managed to recover Y-arm in air locking, and the transmission is up at ~0.6 again which is what we were seeing prior to touching anything on the BS-PRM table, so it looks like the tip-tilt has not gone badly astray... I have also restored the Satellite boxes so that both PRM and SRM have their designated boxes
We would like to establish a system for setting up ADC channels and integrating them into the existing EPICS framework, so that we can gradually switch over channels that are currently handled by the aging slow machines. Otherwise, we will be stuck when they eventually fail. As a preliminary test for this method, we are in the process of setting up an Acromag ADC to read the "Diagnostic" output of the PSL controller. This information will also be useful to monitor the health of the PSL.
Today, we accomplished the following:
The power supply has been turned off for the night.
Detailed elog to follow but summary of todays activities:
Visiting graduate student Teng Zhang from Glasglow received 40m specific safety training yesterday.
[steve, teng, johannes, lydia, gautam]
Depending on how the X arm situation is, we will finish putting back all the heavy doors on Monday and start the pumpdown
GV Edit 11.30pm:
Looks like on Monday, we will look to put the heavy doors on ITMY, ITMX and ETMX chambers, and begin the pumpdown
For the PRM, I aligned it until the arm flashes were maximized and the REFL camera showed a centered spot with dips happening during the arm pops. AS port was more messy since the Michelson alignment wasn't perfect, but the spots were both near the center of the cam and the SRM alignment maximized the wangy fringiness of the image as well as the angry cat meow sounds that the full IFO makes as heard through the DAFI (listening to POX).
On Monday, Osamu should be back and can help with doors and then alignment recovery and locking.
[Gautam, Steve, Johannes]
We put on the remaining heavy doors on the chambers (ITMY, ITMX,ETMX, in this order) this morning. On the ITMY and ETMX tables we placed old OpLev steering mirrors that are clean and baked as witness plates such that may one day provide some insight into dust accumulation on optics.
With the heavy doors on we confirmed that we were still able to lock both IFO arms and used the dither scripts to optimize the alignment. Following that we centered all OpLevs and aligned the X and Y green beams.
We are pumping down. The annuloses are below 10 mTorr
The vacuum gauges are not communicating with the medm so there is no plot available.
The main volume pumping is stopped at P1 = 220 Torr ( New SuperBee 174 Torr ) for overnight.
note: SuperBee was reading 791 Torr at atm and it was not set to 760 !
We ran the scripts to diagonalize the damping matrices using the free swinging data from staurday night/sunday morning. The actual entries used for damping have not been changed. However, we did generate updated matrices for all the main optics (not including the mode cleaner optics, which were not free swinging over the weekend).
The beam splitter that directs light into the MC REFL photodiode has not been replaced; there is still a mirror there. Gautam suggested we wait to replace it until the PSL shutter is open so the beam can be aligned. However, this must be done before going to high power.
GV addendum: What I suggested was to try and recover the arm alignment using the current low power configuration after pumpdown - since we were well aligned just before pumpdown, we should be able to recover this alignment pretty easily at low power. After locking both arms and running the dither align (also center all Oplevs), we can go ahead do the following:
Last night from 8:30 pm to 8:30 am PDT, ETMY UL signal was glitchy again. As of now it seems to have quieted back down, but we pushed on the cables on the board at the Y end to hopefully prevent it from coming back. After doing so it still seems to be behaving well.
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.
THANKS to Chris !
The shutters can be opened with high power.
No communication error message still exist.
I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
Something strange happened to the ITMX osem reading around 4.pm. PDT as shown below.
Also the there was no response of the reading as we adjusted the PITCH and YAW. :(
Note that we restarted the slow machine: c1susaux,c1ausex this afternoon because of the unresponced interface.
We continued to work on the diagonalization scripts today and devised a way of choosing starting parameters that seems to work much better, and is easier to use, than tuning up to 15 parameters by hand per optic.
We still noticed phase problems with ITMY, which appear to be preventing good diagonalization (See Attachment 1). Almost every degree of freedom has a significant imaginary part in the sensing matrix. We looked at the phases of the cross spectra in DDT and saw that indeed, the OSEM signals do not have the appropriate relative phases at the peak frequencies, especially in PIT and YAW (see Attachment 2: the phase at the peak is about 30 degrees when it should be 180). These phases are different for data takes ~24 hours apart, but are still wrong. We also looked at this information for ETMY and saw the correct behavior. We temporarily moved the pitch and yaw sliders for ITMY and looked at the OSEM response on a striptool, and the signals moved in the expected way. Can anyone suggest a reason why this would be happening? Is there another stretch of data (besides this past weekend) which would be good to compare to?
I believe that the UR and LR magnets are stuck. There was no earth quake at 16:18 yesterday. Something had to kick it into this position. See 4days plot
Please advise freeing details.
The pumpdown started at 9-12-2016
The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.
ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.
New pirani and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.
Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is not pumped.
When I restarted c1susaux yesterday, I didn't know that I needed to disable the coil outputs first. So when it came back online, it attempted to damp all the vertex area optics and ITMX got stuck
We should make a note in the Computer Restart Procedures wiki page indicating the importance of disabling the coils before rebooting c1susaux, c1auxex, and c1auxey. Today c1auxey was rebooted properly without incident. If the slider values etc go back to their previous values on their own, is it necessary to do a BURT restore? I tried doing one for c1susaux today and there were some errors for ASC channels, but the alignment sliders went right back to the proper place after reboot yesterday.
Please advise freeing details
Today the main optics were free swinging for several hours, so I attempted diagonalization in vacuum.
IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet
New items in vacuum:
1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ .......... and 2 ruby wirestandoffs.
2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.
3, 50 mm ID green glass baffles at the ends
4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS
5, new pirani, cc gauge and 1.5" right angle valve
South end flow bench and both clean room assembly flow benches measured zero counts for 0.3 and 0.5 micron size particales.
The counting efficiency of 0.5 micron is 100%
0.3 micron particles / cf min
0.5 micron particles / cf min
The PSL HEPA performance was measured at the center of the table with MET ONE #3
We built matrices for ITMY and ETMY by driving one degree of freedom at a time with awggui, while the damping was on. These have been applied to the damping loops.
Pitch:1158085097 Yaw: 1158086537 Pos: 1158089237 Side: 1158087977
Pitch: 1158095897 Yaw: 1158097577 Pos: 1158099377 Side: 1158100817
Today the Y arm was locking fine. The alignment had drifted somewhat so I ran the dither and TRY returned to ~0.8. However, the mode cleaner has been somewhat unstable. It locked many times but usually for only a few minutes. Maybe the alignment or autolocker needs to be adjusted, but I didn't change anything other than playing with the gain sliders (which didn't seem to make it either better or worse).
ITMX is still stuck.
All is not lost. I've stuck and unstuck optics around a half dozen times. Can you please post the zoomed in time series (not trend) from around the time it got stuck? Sometimes the bias sliders have to be toggles to make the bias correct. From the OSEM trend it seems like it got a large Yaw bias. May also try to reseat the satellite box cables and the cable from the coil driver to the cable breakout board in the back of the rack.
Here's the timeseries plots. I've zoomed in to right after the problem- did you want before? We pretty much know what happened: c1susaux was restarted from the crate but the damping was on, so as soon as the machine came back online the damping loops sent a huge signal to the coils. (Also, it seems to be down again. Now we know what to do first before keying the crate.) It seems like both right side magnets are stuck, and this could probably be fixed by moving the yaw slider. Steve advised that we wait for an experienced hand to do so.
susaux is responsible for turning on/off the inputs to the coil driver, but not the actual damping loops. So rebooting susaux only does the same as turning the watchdogs on/off so it shouldn't be a big issue.
Both before and after would be good. We want to see how much bias and how much voltage from the front ends were applied. l1susaux could have put in a huge bias, but NOT a huge force from the damping loops. But I've never seen it put in a huge bias and there's no way to prevent this anyway without disconnecting cables.
I think its much more likely that its a little stuck due to static charge on the rubber EQ stop tips and that we can shake it lose with the damping loops.
ITMX is free, OSEM signals all rougly centered.
This was accomplished by rocking the static alignment (i.e. slow controls) pitch and yaw offsets until the optic broke free. This took a few volts back and forth. At this point, I tried to find a point where the optic seemed to freely swing, and hopefully have signals in all 5 OSEMS. It seemed to be free sometimes but mostly settling into two different stationary states. I realized that it was becoming torqued enough in pitch to be leaning on the top-front or top-back EQ stops. So, I slowly adjusted the pitch from one of these states until it seemed to be swinging a bit on the camera, and three OSEM signals were showing real motion. Then, I slowly adjusted the pitch and yaw alignments to get all OSEMS signals roughly centered at half of their max voltage.
In order to figure out the difference betweent simulated result and measurement, I tried to measuren the electronic noise by following ways as show in attachment 1
1.measure from the satellite box by SR785 at ETMY ,calibrate to counts by divide by 3267.8. while at that conditin, the set up is in suspension.
2. measure after ADC by diagnostics test tools, with set up on table in history and on uspension currently.
3. use the caculated butterfly channel.
the results are shown in attachmemt 2. The overall nosie level are still much higher than simulation.
If we have some data with one of the optics clamped and the open light hitting the PD, or with the OSEMs removed and sitting on the table, that would be useful for evaluating the end-to-end noise of the OSEM circuit. It seems like we probably have that due to the vent work, so please post the times here if you have them.
The ETMX OSEMs have been attached to its Satellite box and plugged in for the last 10 days or so, with the PD exposed to the unobstructed LED. I pulled the spectrum of one of the sensors (mean detrended, I assume this takes care of removing the DC value?). The DQed channels claim to record um (the raw ADC counts are multiplied by a conversion factor of 0.36). For comparison, re-converted the y-axis for the measured curve to counts, and multiplied the total noise curve from the LISO simulation by a factor of 3267.8cts/V (2^16cts/20V) so the Y axis is noise in units of counts/rtHz. At 1Hz, there is more than an order of magnitude difference between the simulation and the measurement which makes me suspect my y-axis conversion, but I think I've done this correctly. Can such a large discrepancy be solely due to thick film resistors?
[ericq, Lydia, Teng]
Brief summary of this afternoon's activities:
Addendum: I had a suspicion that the alignment had moved so much, we were missing the TRX PDs. I misaligned the Y arm, and used AS110 as a proxy for X arm power, as we've done in the past for this kind of thing. Indeed, I could maximize the signal and lock a TM00 mode. Both the high gain PD and QPD in the TRX path are totally dark. This needs realignment on the end table.
Rana suspicious. We had arms locked before pumpdown with beams on Transmon PDs. If they're off now, must be beams are far off on the mirrors. Try A2L to estimate spot positions before walkin the beams too far.
REF33 was removed for taking picture of the bare C30362 InGaAs photodiode per Rana's request. All other rf photodiodes have their glass cover on.
Note: it is back to it's place but this pd will need alignment!
The small steering mirror was completly lose before it was removed.
The misalignment wasn't as bad as I had intially feared; the spot was indeed pretty high on ETMX at first. Both transmon QPDs did need a reasonable amount of steering to center once the dither had centered the beam spots on the optics.
Arms, PRMI and DRMI have all been locked and dither aligned. All oplevs and transmon QPDs have been centered. All AS and REFL photodiodes have been centered.
Green TM00 modes are seen in each arm; I'll do ALS recovery tomorrow.
Good 8 hours
Rana came by and freed ITMX again. I think it shouldn't be a problem for me to free it if it happens again.
In hopes of getting better SNR on the free swing spectra, we kicked all optics at around 7pm. The damping should come back on a little after midnight. ITMX did not get stuck after this kick.
We moved the Acromag and its power supply to the X end, where we connected it to the diagnostic output of the NPRO controller. We renamed the channels to be descriptive of the pin outputs as described in the laser manual. We were able to recover readouts similar to those we found with a multimeter.
We should figure out how to set up the channels on the front end machines: right now they are accessed through a tmux session running on pianosa. Once we are confident in the operation, we will make a box to contain the Acromag and wire connections and move the setup to connect to the PSL controller.
Just a heads up, it looks like the damping came on at around 8:30pm. Not sure why.
I had hoped to do some ALS work, but I realized too late that we loaned our HP analyzer to Andrew. I decided instead to do some ETMX testing.
I have a script running that'll misalign both ETMs and back by about 0.5mrad with half hour rests in between. It'll be done around 6AM.
Seems like the angular position was fairly stable, though there is some change in the ETMX pitch that could be hysterisis or normal drift. I didn't mention it explicity in the previous log, but the misalignment was purely in pitch. I'll give it another shot with a bigger misalginment, and maybe a mix of pitch and yaw.
This afternoon around 2:45, ITMX started ringing up at ~.9Hz for about a minute and then got stuck again. When I noticed this evening, I tried to free it with the alignment sliders but was unable to see any signal on UL or UR. It also looks like the damping for ITMY was turned off at the same time ITMX got stuck (not at the start of its ring up). SRM also has a spike in its motion at this time, and another one minute later that ended up with the LR OSEM at a much higher level, though the mirror does not appear to be stuck. We didn't see any strange behavior from any of the other optics.
Teng and I were working on diagnosing a problem with the ITMY UL whitening, but by the time we disconnected any applicable cables, the damping for ITMY was already off. Later we unplugged the ITMX PD whitening cables after verifying that the ITMX damping was also already off. This problem may have occured earlier, while Teng, Eric, and I were examining and pushing in the cables at 1X5 without unplugging anything.
We found that the reason for the bad phase on the ITMY free swing data is because the whitening filter for UL is not being properly turned on. We are in the process of investigating the source of this problem. Right now all the cables to the PD whitening boxes in 1X5 are switched between ITMY and ITMX.
The earth quake shook ITMX free for a short while.
When we plugged in the back cables yesterday on the whitening boxes after switching them, two of the ITMX PDMon channels (UR and LR) got stuck at 0. This caused me to believe ITMX was still stuck even when it was freed. However, it was left in a stuck state overnight and freed again today after this issue was discovered. The alignment sliders have been set to 0 as a safety net to keep ITMX from getting stuck again if c1susaux is restarted again. We switched the cables back and the problem was still there.
The ITMY UL whitening filter problem, which the cables were originally switched to diagnose, was also still there. Ericq suggested we turn off all the whitening filters in order to get diagonalization data that would not show a phase difference between coils. We ran the diagonalization again with all the dewhitening filters off and got much cleaner results, with no visible cross-coupling peaks remaining between the degrees of freedom (see attachemnt 1). We did not apply this matrix to the damping, however, because there are elements which have the wrong sign compared to the ideal matrix. Significant adjustments to the output matrix will probably need to be made if this result is to be used. We also verified that the phase problem had been solved in DTT, where we saw the same sign discrepancies as in the matrix below.
Damping can be turned back on, using the old, non-diagonalized matrix currently in effect. There is enough free swing data to diagonalize ITMY now, so feel free to mess with it.
Matrix (wrong signs red, suspiciously small elements orange):
pit yaw pos side butt
UL 1.633 0.138 1.224 0.136 0.984
UR -0.202 -1.768 1.179 0.132 -1.028
LR -2.000 0.094 0.776 0.107 1.001
LL -0.165 2.000 0.821 0.111 -0.987
SD 0.900 1.131 -1.708 1.000 -0.107
PMC was terribly misaligned. The PMCR camera seems to have drifted somewhat off target too, but I didn't touch it.
Realigned ITMX for the nth time today.
Finding ALSY beatnote was easy, ALSX eludes me. I did a rough one-point realignment on the X beat PD which is usually enough, but it's probably been long enough that near/far field alignmnet is neccesary.
ALSY noise is mostly nominal, but there is a large 3Hz peak that is visible in the spot motion, and also modulates the beat amplitude by multiple dBs.
It looked to me that the ETMY oplev spot was moving too much, which led me to measure the oplev OLGs. There is some wierd inter-loop interference going on between OLPIT and OLYAW. With both on (whether OSEM damping is on or off, so input matrix shenanigans can't be to blame) there is a very shallow "notch" at around 4.5Hz, which leads to very little phase at 3Hz, and thus tons of control noise. Turning the OL loop not being measured off makes this dip go away, but the overall phase is still signfinicantly less than we should have. I'm not sure why. I'll just show the PIT plot, but things look pretty much the same for YAW.
I did some more ETMX tests. Locked arm, raised the servo output limit to 15k, then increased the gain to make the loop unstable. I saw the SUS LSC signals go up to tens of thousands of counts when the unlock happened. I did this a dozen times or so, and every time the ETM settled in the same angular position according to the oplev.
Right now, another hysteresis script is running, misaliging in pitch and yaw. Amplitude 1V in each direction. So far, everything is stable after three on/off cycles.
With the WFS and OL, we never have figured out a good way to separate pit and yaw. Need to figure out a reference for up/down and then align everything to it: quad matrix + SUS output matrix
Motivated by the strange pitch/yaw coupling behavior we ran into while doing diagonalization, we looked at the oplev pitch and yaw free swing spectra for all 4 test masses (see attachment 1). We saw the same behavior there: At the peak frequencies for the angular degress of freedom, the oplevs saw significant contributions from both pitch and yaw. We also examined the phase between pitch and yaw at these peaks and found that consistently, pitch and yaw were in phase at one of the resonance frequencies and out of phase at the other (ignoring the pos and side peaks).
This corresponds physically to angular motion about some axis that is diagonal, ie not perfectly vertical or horizontal. If we trust the oplev calibration, and Eric says that we do, then the angle of this axis of rotation with the horizontal (pitch axis) is
Where Y and P are yaw and pitch ASD values. This will always give an angle between 0 and 90 degrees; which quadrant the axis of rotation occupies can be dermined by looking at the phase between pitch and yaw at the same frequencies. 0 phase means that the axis of rotation lies somewhere less than 90 degrees counterclockwise from the horizontal as viewed from the AR face of the optic, and a phase of 180 degrees means the axis is clockwise from horizontal (see attachment 2). Qualitatively, these features show up the same way for segments of data taken at different times. In order to get some quantitative sense of the error in these angles, we found them using spectrogram values with a bandwidth of 0.02 Hz averaged over 4000 seconds.
Results (all numbers in degrees unless otherwise specified):
peak 1 ( 0.692 Hz):
ptich/yaw phase: -179.181
peak 2 ( 0.736 Hz):
pitch/yaw phase: 0.0123677
peak 1 ( 0.502 Hz):
ptich/yaw phase: -179.471
peak 2 ( 0.688 Hz):
pitch/yaw phase: -0.43991
peak 1 ( 0.73 Hz):
ptich/yaw phase: -0.227034
peak 2 ( 0.85 Hz):
pitch/yaw phase: -179.856
peak 1 ( 0.724 Hz):
ptich/yaw phase: 6.03312
peak 2 ( 0.844 Hz):
pitch/yaw phase: -176.838
ETMY and ITMX both show a more significant (~4x) contribution from pitch on one peak, and from yaw on the other. This is reflected in the fact that they each have one angle somewhat close to 0 (below 30 degrees) and one close to 90 (above 60 degrees). The other two test masses don't follow this rule, meaning that the 2 angular frequency peaks do not correspond to pitch and yaw straightforwardly.
Also, besides ITMX, the axes of rotation are at least several degrees away from being perpendicular to each other.