When we plugged in the back cables yesterday on the whitening boxes after switching them, two of the ITMX PDMon channels (UR and LR) got stuck at 0. This caused me to believe ITMX was still stuck even when it was freed. However, it was left in a stuck state overnight and freed again today after this issue was discovered. The alignment sliders have been set to 0 as a safety net to keep ITMX from getting stuck again if c1susaux is restarted again. We switched the cables back and the problem was still there.
The ITMY UL whitening filter problem, which the cables were originally switched to diagnose, was also still there. Ericq suggested we turn off all the whitening filters in order to get diagonalization data that would not show a phase difference between coils. We ran the diagonalization again with all the dewhitening filters off and got much cleaner results, with no visible cross-coupling peaks remaining between the degrees of freedom (see attachemnt 1). We did not apply this matrix to the damping, however, because there are elements which have the wrong sign compared to the ideal matrix. Significant adjustments to the output matrix will probably need to be made if this result is to be used. We also verified that the phase problem had been solved in DTT, where we saw the same sign discrepancies as in the matrix below.
Damping can be turned back on, using the old, non-diagonalized matrix currently in effect. There is enough free swing data to diagonalize ITMY now, so feel free to mess with it.
Matrix (wrong signs red, suspiciously small elements orange):
pit yaw pos side butt
UL 1.633 0.138 1.224 0.136 0.984
UR -0.202 -1.768 1.179 0.132 -1.028
LR -2.000 0.094 0.776 0.107 1.001
LL -0.165 2.000 0.821 0.111 -0.987
SD 0.900 1.131 -1.708 1.000 -0.107
The earth quake shook ITMX free for a short while.
This afternoon around 2:45, ITMX started ringing up at ~.9Hz for about a minute and then got stuck again. When I noticed this evening, I tried to free it with the alignment sliders but was unable to see any signal on UL or UR. It also looks like the damping for ITMY was turned off at the same time ITMX got stuck (not at the start of its ring up). SRM also has a spike in its motion at this time, and another one minute later that ended up with the LR OSEM at a much higher level, though the mirror does not appear to be stuck. We didn't see any strange behavior from any of the other optics.
Teng and I were working on diagnosing a problem with the ITMY UL whitening, but by the time we disconnected any applicable cables, the damping for ITMY was already off. Later we unplugged the ITMX PD whitening cables after verifying that the ITMX damping was also already off. This problem may have occured earlier, while Teng, Eric, and I were examining and pushing in the cables at 1X5 without unplugging anything.
We found that the reason for the bad phase on the ITMY free swing data is because the whitening filter for UL is not being properly turned on. We are in the process of investigating the source of this problem. Right now all the cables to the PD whitening boxes in 1X5 are switched between ITMY and ITMX.
Seems like the angular position was fairly stable, though there is some change in the ETMX pitch that could be hysterisis or normal drift. I didn't mention it explicity in the previous log, but the misalignment was purely in pitch. I'll give it another shot with a bigger misalginment, and maybe a mix of pitch and yaw.
I had hoped to do some ALS work, but I realized too late that we loaned our HP analyzer to Andrew. I decided instead to do some ETMX testing.
I have a script running that'll misalign both ETMs and back by about 0.5mrad with half hour rests in between. It'll be done around 6AM.
Just a heads up, it looks like the damping came on at around 8:30pm. Not sure why.
We moved the Acromag and its power supply to the X end, where we connected it to the diagnostic output of the NPRO controller. We renamed the channels to be descriptive of the pin outputs as described in the laser manual. We were able to recover readouts similar to those we found with a multimeter.
We should figure out how to set up the channels on the front end machines: right now they are accessed through a tmux session running on pianosa. Once we are confident in the operation, we will make a box to contain the Acromag and wire connections and move the setup to connect to the PSL controller.
Rana came by and freed ITMX again. I think it shouldn't be a problem for me to free it if it happens again.
In hopes of getting better SNR on the free swing spectra, we kicked all optics at around 7pm. The damping should come back on a little after midnight. ITMX did not get stuck after this kick.
Good 8 hours
The misalignment wasn't as bad as I had intially feared; the spot was indeed pretty high on ETMX at first. Both transmon QPDs did need a reasonable amount of steering to center once the dither had centered the beam spots on the optics.
Arms, PRMI and DRMI have all been locked and dither aligned. All oplevs and transmon QPDs have been centered. All AS and REFL photodiodes have been centered.
Green TM00 modes are seen in each arm; I'll do ALS recovery tomorrow.
REF33 was removed for taking picture of the bare C30362 InGaAs photodiode per Rana's request. All other rf photodiodes have their glass cover on.
Note: it is back to it's place but this pd will need alignment!
The small steering mirror was completly lose before it was removed.
Rana suspicious. We had arms locked before pumpdown with beams on Transmon PDs. If they're off now, must be beams are far off on the mirrors. Try A2L to estimate spot positions before walkin the beams too far.
[ericq, Lydia, Teng]
Brief summary of this afternoon's activities:
Addendum: I had a suspicion that the alignment had moved so much, we were missing the TRX PDs. I misaligned the Y arm, and used AS110 as a proxy for X arm power, as we've done in the past for this kind of thing. Indeed, I could maximize the signal and lock a TM00 mode. Both the high gain PD and QPD in the TRX path are totally dark. This needs realignment on the end table.
In order to figure out the difference betweent simulated result and measurement, I tried to measuren the electronic noise by following ways as show in attachment 1
1.measure from the satellite box by SR785 at ETMY ,calibrate to counts by divide by 3267.8. while at that conditin, the set up is in suspension.
2. measure after ADC by diagnostics test tools, with set up on table in history and on uspension currently.
3. use the caculated butterfly channel.
the results are shown in attachmemt 2. The overall nosie level are still much higher than simulation.
If we have some data with one of the optics clamped and the open light hitting the PD, or with the OSEMs removed and sitting on the table, that would be useful for evaluating the end-to-end noise of the OSEM circuit. It seems like we probably have that due to the vent work, so please post the times here if you have them.
The ETMX OSEMs have been attached to its Satellite box and plugged in for the last 10 days or so, with the PD exposed to the unobstructed LED. I pulled the spectrum of one of the sensors (mean detrended, I assume this takes care of removing the DC value?). The DQed channels claim to record um (the raw ADC counts are multiplied by a conversion factor of 0.36). For comparison, re-converted the y-axis for the measured curve to counts, and multiplied the total noise curve from the LISO simulation by a factor of 3267.8cts/V (2^16cts/20V) so the Y axis is noise in units of counts/rtHz. At 1Hz, there is more than an order of magnitude difference between the simulation and the measurement which makes me suspect my y-axis conversion, but I think I've done this correctly. Can such a large discrepancy be solely due to thick film resistors?
ITMX is free, OSEM signals all rougly centered.
This was accomplished by rocking the static alignment (i.e. slow controls) pitch and yaw offsets until the optic broke free. This took a few volts back and forth. At this point, I tried to find a point where the optic seemed to freely swing, and hopefully have signals in all 5 OSEMS. It seemed to be free sometimes but mostly settling into two different stationary states. I realized that it was becoming torqued enough in pitch to be leaning on the top-front or top-back EQ stops. So, I slowly adjusted the pitch from one of these states until it seemed to be swinging a bit on the camera, and three OSEM signals were showing real motion. Then, I slowly adjusted the pitch and yaw alignments to get all OSEMS signals roughly centered at half of their max voltage.
susaux is responsible for turning on/off the inputs to the coil driver, but not the actual damping loops. So rebooting susaux only does the same as turning the watchdogs on/off so it shouldn't be a big issue.
Both before and after would be good. We want to see how much bias and how much voltage from the front ends were applied. l1susaux could have put in a huge bias, but NOT a huge force from the damping loops. But I've never seen it put in a huge bias and there's no way to prevent this anyway without disconnecting cables.
I think its much more likely that its a little stuck due to static charge on the rubber EQ stop tips and that we can shake it lose with the damping loops.
Here's the timeseries plots. I've zoomed in to right after the problem- did you want before? We pretty much know what happened: c1susaux was restarted from the crate but the damping was on, so as soon as the machine came back online the damping loops sent a huge signal to the coils. (Also, it seems to be down again. Now we know what to do first before keying the crate.) It seems like both right side magnets are stuck, and this could probably be fixed by moving the yaw slider. Steve advised that we wait for an experienced hand to do so.
All is not lost. I've stuck and unstuck optics around a half dozen times. Can you please post the zoomed in time series (not trend) from around the time it got stuck? Sometimes the bias sliders have to be toggles to make the bias correct. From the OSEM trend it seems like it got a large Yaw bias. May also try to reseat the satellite box cables and the cable from the coil driver to the cable breakout board in the back of the rack.
Today the Y arm was locking fine. The alignment had drifted somewhat so I ran the dither and TRY returned to ~0.8. However, the mode cleaner has been somewhat unstable. It locked many times but usually for only a few minutes. Maybe the alignment or autolocker needs to be adjusted, but I didn't change anything other than playing with the gain sliders (which didn't seem to make it either better or worse).
ITMX is still stuck.
We built matrices for ITMY and ETMY by driving one degree of freedom at a time with awggui, while the damping was on. These have been applied to the damping loops.
Pitch:1158085097 Yaw: 1158086537 Pos: 1158089237 Side: 1158087977
Pitch: 1158095897 Yaw: 1158097577 Pos: 1158099377 Side: 1158100817
South end flow bench and both clean room assembly flow benches measured zero counts for 0.3 and 0.5 micron size particales.
The counting efficiency of 0.5 micron is 100%
0.3 micron particles / cf min
0.5 micron particles / cf min
The PSL HEPA performance was measured at the center of the table with MET ONE #3
IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet
New items in vacuum:
1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ .......... and 2 ruby wirestandoffs.
2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.
3, 50 mm ID green glass baffles at the ends
4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS
5, new pirani, cc gauge and 1.5" right angle valve
The pumpdown started at 9-12-2016
The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.
ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.
New pirani and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.
Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is not pumped.
THANKS to Chris !
The shutters can be opened with high power.
No communication error message still exist.
I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.
Today the main optics were free swinging for several hours, so I attempted diagonalization in vacuum.
When I restarted c1susaux yesterday, I didn't know that I needed to disable the coil outputs first. So when it came back online, it attempted to damp all the vertex area optics and ITMX got stuck
We should make a note in the Computer Restart Procedures wiki page indicating the importance of disabling the coils before rebooting c1susaux, c1auxex, and c1auxey. Today c1auxey was rebooted properly without incident. If the slider values etc go back to their previous values on their own, is it necessary to do a BURT restore? I tried doing one for c1susaux today and there were some errors for ASC channels, but the alignment sliders went right back to the proper place after reboot yesterday.
I believe that the UR and LR magnets are stuck. There was no earth quake at 16:18 yesterday. Something had to kick it into this position. See 4days plot
Please advise freeing details
Please advise freeing details.
Something strange happened to the ITMX osem reading around 4.pm. PDT as shown below.
Also the there was no response of the reading as we adjusted the PITCH and YAW. :(
Note that we restarted the slow machine: c1susaux,c1ausex this afternoon because of the unresponced interface.
We continued to work on the diagonalization scripts today and devised a way of choosing starting parameters that seems to work much better, and is easier to use, than tuning up to 15 parameters by hand per optic.
We still noticed phase problems with ITMY, which appear to be preventing good diagonalization (See Attachment 1). Almost every degree of freedom has a significant imaginary part in the sensing matrix. We looked at the phases of the cross spectra in DDT and saw that indeed, the OSEM signals do not have the appropriate relative phases at the peak frequencies, especially in PIT and YAW (see Attachment 2: the phase at the peak is about 30 degrees when it should be 180). These phases are different for data takes ~24 hours apart, but are still wrong. We also looked at this information for ETMY and saw the correct behavior. We temporarily moved the pitch and yaw sliders for ITMY and looked at the OSEM response on a striptool, and the signals moved in the expected way. Can anyone suggest a reason why this would be happening? Is there another stretch of data (besides this past weekend) which would be good to compare to?
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
For the ITMY, I squished together the cables which are in the 'Cable Interface Board' which lives in the rack. This thing takes the 64 pin IDC from the satellite module and converts it into 2 D-sub connectors to go to the PD whitening board and the coil driver board. Lets see if the ITMY OSEM glitches change character overnight.
Last night from 8:30 pm to 8:30 am PDT, ETMY UL signal was glitchy again. As of now it seems to have quieted back down, but we pushed on the cables on the board at the Y end to hopefully prevent it from coming back. After doing so it still seems to be behaving well.
The beam splitter that directs light into the MC REFL photodiode has not been replaced; there is still a mirror there. Gautam suggested we wait to replace it until the PSL shutter is open so the beam can be aligned. However, this must be done before going to high power.
GV addendum: What I suggested was to try and recover the arm alignment using the current low power configuration after pumpdown - since we were well aligned just before pumpdown, we should be able to recover this alignment pretty easily at low power. After locking both arms and running the dither align (also center all Oplevs), we can go ahead do the following:
We ran the scripts to diagonalize the damping matrices using the free swinging data from staurday night/sunday morning. The actual entries used for damping have not been changed. However, we did generate updated matrices for all the main optics (not including the mode cleaner optics, which were not free swinging over the weekend).
We are pumping down. The annuloses are below 10 mTorr
The vacuum gauges are not communicating with the medm so there is no plot available.
The main volume pumping is stopped at P1 = 220 Torr ( New SuperBee 174 Torr ) for overnight.
note: SuperBee was reading 791 Torr at atm and it was not set to 760 !
[Gautam, Steve, Johannes]
We put on the remaining heavy doors on the chambers (ITMY, ITMX,ETMX, in this order) this morning. On the ITMY and ETMX tables we placed old OpLev steering mirrors that are clean and baked as witness plates such that may one day provide some insight into dust accumulation on optics.
With the heavy doors on we confirmed that we were still able to lock both IFO arms and used the dither scripts to optimize the alignment. Following that we centered all OpLevs and aligned the X and Y green beams.
For the PRM, I aligned it until the arm flashes were maximized and the REFL camera showed a centered spot with dips happening during the arm pops. AS port was more messy since the Michelson alignment wasn't perfect, but the spots were both near the center of the cam and the SRM alignment maximized the wangy fringiness of the image as well as the angry cat meow sounds that the full IFO makes as heard through the DAFI (listening to POX).
On Monday, Osamu should be back and can help with doors and then alignment recovery and locking.
[steve, teng, johannes, lydia, gautam]
Depending on how the X arm situation is, we will finish putting back all the heavy doors on Monday and start the pumpdown
GV Edit 11.30pm:
Looks like on Monday, we will look to put the heavy doors on ITMY, ITMX and ETMX chambers, and begin the pumpdown
Visiting graduate student Teng Zhang from Glasglow received 40m specific safety training yesterday.
Detailed elog to follow but summary of todays activities:
We would like to establish a system for setting up ADC channels and integrating them into the existing EPICS framework, so that we can gradually switch over channels that are currently handled by the aging slow machines. Otherwise, we will be stuck when they eventually fail. As a preliminary test for this method, we are in the process of setting up an Acromag ADC to read the "Diagnostic" output of the PSL controller. This information will also be useful to monitor the health of the PSL.
Today, we accomplished the following:
The power supply has been turned off for the night.
Edit 7.30pm: I have managed to recover Y-arm in air locking, and the transmission is up at ~0.6 again which is what we were seeing prior to touching anything on the BS-PRM table, so it looks like the tip-tilt has not gone badly astray... I have also restored the Satellite boxes so that both PRM and SRM have their designated boxes
The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
We peeked into the BS-PRM chamber via the ITMX chamber to see if we could shed any light on this situation. It's hard to get a picture that is in focus, but it looks quite clear that the LR LED (in the lower left when viewed from the HR side) isn't anywhere near as bright as the rest (see Attachment #1). Various hypothesis include failed LED / piece of Al foil blocking the LED / teflon aperture slipped over the LED. But looks like we can't solve this without opening up the BS-PRM chamber. The plan tomorrow is to open up the chamber, pull out the problematic coil. Once we have a better idea of what is going wrong, we can decide what the appropriate course of action is - replace the OSEM or something else.
As part of the diagnosis, I switched the PRM and SRM satellite boxes earlier today evening around 6pm. They remain in this switched state for now.
Steve, we plan to take the BS-PRM heavy door off tomorrow morning.
[Teng, Johannes, Lydia, gautam]
In the morning, Steve will start opening the north BS door so that we can enter to inspect the PRM LR OSEM.
I looked at the PRM free swing spectra. The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
Some of the frequencies have changed between the 2011 in-vac measurement and our 2016 in-air measurement, but that seems within usual parameters.