[ericq, Lydia, Teng]
Brief summary of this afternoon's activities:
Addendum: I had a suspicion that the alignment had moved so much, we were missing the TRX PDs. I misaligned the Y arm, and used AS110 as a proxy for X arm power, as we've done in the past for this kind of thing. Indeed, I could maximize the signal and lock a TM00 mode. Both the high gain PD and QPD in the TRX path are totally dark. This needs realignment on the end table.
In order to figure out the difference betweent simulated result and measurement, I tried to measuren the electronic noise by following ways as show in attachment 1
1.measure from the satellite box by SR785 at ETMY ,calibrate to counts by divide by 3267.8. while at that conditin, the set up is in suspension.
2. measure after ADC by diagnostics test tools, with set up on table in history and on uspension currently.
3. use the caculated butterfly channel.
the results are shown in attachmemt 2. The overall nosie level are still much higher than simulation.
If we have some data with one of the optics clamped and the open light hitting the PD, or with the OSEMs removed and sitting on the table, that would be useful for evaluating the end-to-end noise of the OSEM circuit. It seems like we probably have that due to the vent work, so please post the times here if you have them.
The ETMX OSEMs have been attached to its Satellite box and plugged in for the last 10 days or so, with the PD exposed to the unobstructed LED. I pulled the spectrum of one of the sensors (mean detrended, I assume this takes care of removing the DC value?). The DQed channels claim to record um (the raw ADC counts are multiplied by a conversion factor of 0.36). For comparison, re-converted the y-axis for the measured curve to counts, and multiplied the total noise curve from the LISO simulation by a factor of 3267.8cts/V (2^16cts/20V) so the Y axis is noise in units of counts/rtHz. At 1Hz, there is more than an order of magnitude difference between the simulation and the measurement which makes me suspect my y-axis conversion, but I think I've done this correctly. Can such a large discrepancy be solely due to thick film resistors?
ITMX is free, OSEM signals all rougly centered.
This was accomplished by rocking the static alignment (i.e. slow controls) pitch and yaw offsets until the optic broke free. This took a few volts back and forth. At this point, I tried to find a point where the optic seemed to freely swing, and hopefully have signals in all 5 OSEMS. It seemed to be free sometimes but mostly settling into two different stationary states. I realized that it was becoming torqued enough in pitch to be leaning on the top-front or top-back EQ stops. So, I slowly adjusted the pitch from one of these states until it seemed to be swinging a bit on the camera, and three OSEM signals were showing real motion. Then, I slowly adjusted the pitch and yaw alignments to get all OSEMS signals roughly centered at half of their max voltage.
susaux is responsible for turning on/off the inputs to the coil driver, but not the actual damping loops. So rebooting susaux only does the same as turning the watchdogs on/off so it shouldn't be a big issue.
Both before and after would be good. We want to see how much bias and how much voltage from the front ends were applied. l1susaux could have put in a huge bias, but NOT a huge force from the damping loops. But I've never seen it put in a huge bias and there's no way to prevent this anyway without disconnecting cables.
I think its much more likely that its a little stuck due to static charge on the rubber EQ stop tips and that we can shake it lose with the damping loops.
Here's the timeseries plots. I've zoomed in to right after the problem- did you want before? We pretty much know what happened: c1susaux was restarted from the crate but the damping was on, so as soon as the machine came back online the damping loops sent a huge signal to the coils. (Also, it seems to be down again. Now we know what to do first before keying the crate.) It seems like both right side magnets are stuck, and this could probably be fixed by moving the yaw slider. Steve advised that we wait for an experienced hand to do so.
All is not lost. I've stuck and unstuck optics around a half dozen times. Can you please post the zoomed in time series (not trend) from around the time it got stuck? Sometimes the bias sliders have to be toggles to make the bias correct. From the OSEM trend it seems like it got a large Yaw bias. May also try to reseat the satellite box cables and the cable from the coil driver to the cable breakout board in the back of the rack.
Today the Y arm was locking fine. The alignment had drifted somewhat so I ran the dither and TRY returned to ~0.8. However, the mode cleaner has been somewhat unstable. It locked many times but usually for only a few minutes. Maybe the alignment or autolocker needs to be adjusted, but I didn't change anything other than playing with the gain sliders (which didn't seem to make it either better or worse).
ITMX is still stuck.
We built matrices for ITMY and ETMY by driving one degree of freedom at a time with awggui, while the damping was on. These have been applied to the damping loops.
Pitch:1158085097 Yaw: 1158086537 Pos: 1158089237 Side: 1158087977
Pitch: 1158095897 Yaw: 1158097577 Pos: 1158099377 Side: 1158100817
South end flow bench and both clean room assembly flow benches measured zero counts for 0.3 and 0.5 micron size particales.
The counting efficiency of 0.5 micron is 100%
0.3 micron particles / cf min
0.5 micron particles / cf min
The PSL HEPA performance was measured at the center of the table with MET ONE #3
IFO pressure 3.7e-5 Torr at new cold cathode InstruTech - Hornet
New items in vacuum:
1, ETMX sus tower with new baked sus wire, EP30-2 epoxied magnets, same at different locations also........ .......... and 2 ruby wirestandoffs.
2, First Contact cleaned arm test masses only. This technic was a 1st time use in our vacuum system.
3, 50 mm ID green glass baffles at the ends
4, witness mirrors at ETMX and ITMY (old oplev mirrors) We observed a very dusty system: sides of optics, towers and tables were wiped. Hepa tents used at Y arm and BS
5, new pirani, cc gauge and 1.5" right angle valve
The pumpdown started at 9-12-2016
The IFO is at 5e-5 Torr vacuum normal after 73 day at atm.
ITMX needed to be freed and ETMY-UL is still misbehaving occasionally.
New pirani and cold cathode gauges added at this vent. They were baked at 100 C for 6 hrs under vacuum.
Go to the Vac Rack to read IFO pressure from the gauge itself when Vac. Monitor is blank as it is now !
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is not pumped.
THANKS to Chris !
The shutters can be opened with high power.
No communication error message still exist.
I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.
Today the main optics were free swinging for several hours, so I attempted diagonalization in vacuum.
When I restarted c1susaux yesterday, I didn't know that I needed to disable the coil outputs first. So when it came back online, it attempted to damp all the vertex area optics and ITMX got stuck
We should make a note in the Computer Restart Procedures wiki page indicating the importance of disabling the coils before rebooting c1susaux, c1auxex, and c1auxey. Today c1auxey was rebooted properly without incident. If the slider values etc go back to their previous values on their own, is it necessary to do a BURT restore? I tried doing one for c1susaux today and there were some errors for ASC channels, but the alignment sliders went right back to the proper place after reboot yesterday.
I believe that the UR and LR magnets are stuck. There was no earth quake at 16:18 yesterday. Something had to kick it into this position. See 4days plot
Please advise freeing details
Please advise freeing details.
Something strange happened to the ITMX osem reading around 4.pm. PDT as shown below.
Also the there was no response of the reading as we adjusted the PITCH and YAW. :(
Note that we restarted the slow machine: c1susaux,c1ausex this afternoon because of the unresponced interface.
We continued to work on the diagonalization scripts today and devised a way of choosing starting parameters that seems to work much better, and is easier to use, than tuning up to 15 parameters by hand per optic.
We still noticed phase problems with ITMY, which appear to be preventing good diagonalization (See Attachment 1). Almost every degree of freedom has a significant imaginary part in the sensing matrix. We looked at the phases of the cross spectra in DDT and saw that indeed, the OSEM signals do not have the appropriate relative phases at the peak frequencies, especially in PIT and YAW (see Attachment 2: the phase at the peak is about 30 degrees when it should be 180). These phases are different for data takes ~24 hours apart, but are still wrong. We also looked at this information for ETMY and saw the correct behavior. We temporarily moved the pitch and yaw sliders for ITMY and looked at the OSEM response on a striptool, and the signals moved in the expected way. Can anyone suggest a reason why this would be happening? Is there another stretch of data (besides this past weekend) which would be good to compare to?
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
For the ITMY, I squished together the cables which are in the 'Cable Interface Board' which lives in the rack. This thing takes the 64 pin IDC from the satellite module and converts it into 2 D-sub connectors to go to the PD whitening board and the coil driver board. Lets see if the ITMY OSEM glitches change character overnight.
Last night from 8:30 pm to 8:30 am PDT, ETMY UL signal was glitchy again. As of now it seems to have quieted back down, but we pushed on the cables on the board at the Y end to hopefully prevent it from coming back. After doing so it still seems to be behaving well.
The beam splitter that directs light into the MC REFL photodiode has not been replaced; there is still a mirror there. Gautam suggested we wait to replace it until the PSL shutter is open so the beam can be aligned. However, this must be done before going to high power.
GV addendum: What I suggested was to try and recover the arm alignment using the current low power configuration after pumpdown - since we were well aligned just before pumpdown, we should be able to recover this alignment pretty easily at low power. After locking both arms and running the dither align (also center all Oplevs), we can go ahead do the following:
We ran the scripts to diagonalize the damping matrices using the free swinging data from staurday night/sunday morning. The actual entries used for damping have not been changed. However, we did generate updated matrices for all the main optics (not including the mode cleaner optics, which were not free swinging over the weekend).
We are pumping down. The annuloses are below 10 mTorr
The vacuum gauges are not communicating with the medm so there is no plot available.
The main volume pumping is stopped at P1 = 220 Torr ( New SuperBee 174 Torr ) for overnight.
note: SuperBee was reading 791 Torr at atm and it was not set to 760 !
[Gautam, Steve, Johannes]
We put on the remaining heavy doors on the chambers (ITMY, ITMX,ETMX, in this order) this morning. On the ITMY and ETMX tables we placed old OpLev steering mirrors that are clean and baked as witness plates such that may one day provide some insight into dust accumulation on optics.
With the heavy doors on we confirmed that we were still able to lock both IFO arms and used the dither scripts to optimize the alignment. Following that we centered all OpLevs and aligned the X and Y green beams.
For the PRM, I aligned it until the arm flashes were maximized and the REFL camera showed a centered spot with dips happening during the arm pops. AS port was more messy since the Michelson alignment wasn't perfect, but the spots were both near the center of the cam and the SRM alignment maximized the wangy fringiness of the image as well as the angry cat meow sounds that the full IFO makes as heard through the DAFI (listening to POX).
On Monday, Osamu should be back and can help with doors and then alignment recovery and locking.
[steve, teng, johannes, lydia, gautam]
Depending on how the X arm situation is, we will finish putting back all the heavy doors on Monday and start the pumpdown
GV Edit 11.30pm:
Looks like on Monday, we will look to put the heavy doors on ITMY, ITMX and ETMX chambers, and begin the pumpdown
Visiting graduate student Teng Zhang from Glasglow received 40m specific safety training yesterday.
Detailed elog to follow but summary of todays activities:
We would like to establish a system for setting up ADC channels and integrating them into the existing EPICS framework, so that we can gradually switch over channels that are currently handled by the aging slow machines. Otherwise, we will be stuck when they eventually fail. As a preliminary test for this method, we are in the process of setting up an Acromag ADC to read the "Diagnostic" output of the PSL controller. This information will also be useful to monitor the health of the PSL.
Today, we accomplished the following:
The power supply has been turned off for the night.
Edit 7.30pm: I have managed to recover Y-arm in air locking, and the transmission is up at ~0.6 again which is what we were seeing prior to touching anything on the BS-PRM table, so it looks like the tip-tilt has not gone badly astray... I have also restored the Satellite boxes so that both PRM and SRM have their designated boxes
The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
We peeked into the BS-PRM chamber via the ITMX chamber to see if we could shed any light on this situation. It's hard to get a picture that is in focus, but it looks quite clear that the LR LED (in the lower left when viewed from the HR side) isn't anywhere near as bright as the rest (see Attachment #1). Various hypothesis include failed LED / piece of Al foil blocking the LED / teflon aperture slipped over the LED. But looks like we can't solve this without opening up the BS-PRM chamber. The plan tomorrow is to open up the chamber, pull out the problematic coil. Once we have a better idea of what is going wrong, we can decide what the appropriate course of action is - replace the OSEM or something else.
As part of the diagnosis, I switched the PRM and SRM satellite boxes earlier today evening around 6pm. They remain in this switched state for now.
Steve, we plan to take the BS-PRM heavy door off tomorrow morning.
[Teng, Johannes, Lydia, gautam]
In the morning, Steve will start opening the north BS door so that we can enter to inspect the PRM LR OSEM.
I looked at the PRM free swing spectra. The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
Some of the frequencies have changed between the 2011 in-vac measurement and our 2016 in-air measurement, but that seems within usual parameters.
The attached PDF shows the output noise of the satellite amp. This was calculated using 'osempd.fil' in the 40m/LISO GitLab repo.
The mean voltage output is ~1 Vdc, which corresponds to a current with a shot noise level of 100 nV/rHz on this plot. So the opamp current noise dominates below 1 Hz as long as the OSEM LED output is indeed quantum limited down to 0.1 Hz. Sounds highly implausible.
To convert into meters, we divide by the OSEM conversion factor of ~1.6 V/mm, so the shot noise equivalent would be ~1e-10 m/rHz above 1 Hz.
After adding the sat amp to the 40m DCC tree (D1600348), I notice that not only is the PD readout not built for low noise, neither is the LED drive. The noise should be dominated by the voltage noise of the LT1031 voltage reference. This has a noise of ~500 nV/rHz at 1 Hz. That corresponds to an equivalent current noise through the LED of 25 mA * (500e-9 / 10) ~ 1 nA/rHz. Or ~45 nV/rHz at the sat amp output. This would be OK as long as everything behaves ideally. BUT, we have thick film (i.e. black surface mount) resistors on the LED drive so we'll have to measure it to make sure.
Also, why is the OSEM LED included in the feedback loop of the driver? It means disconnecting the cable from the sat amp makes the driver go unstable probably. I think one concept is that including the device in the feedback loop makes it so that any EMI picked up in the cabling, etc. gets cancelled out by the opamp. But this then requires that we test each driver to make sure it doesn't oscillate when driving the long cable.
Hi 40m people,
As Rana is saying, the bounce mode does not matter, or we cannot do anything. Generally speaking, the bounce mode cannot be damped by the setting of 40m SUS. Some tweak techniques may damp a bounce mode by res-gain or something, but it is not a proper way, I think.
This is also that Rana is already saying that the important thing is to find a good direction of OSEM to hit the LED beam to the magnet. Even if the magnet is not located at the center of OSEM hole, still you can find the optimal orientation of OSEM to hit the LED beam to the center of magnet by rotating the OSEM.
I know only an old document of T040054 that Shihori summarized how to adjust the matrix at the 40m. Too bad input/output matrix may introduce some troubles, but even roughly adjusted matrix should be still fine.
I will be at Caltech on 12-14 of September. If I can help something, I am willing to work with you!
In November of 2010, Valera Frolov (LLO), investigated our satellite amplifiers and made some recommendations about how to increase the SNR.
In light of the recent issues, we ought to fix up one of the spares into this state and swap it in for the ITMY's funky box.
The sat amp schematic is (D961289). It has several versions. Our spare is labeled as version D (not a choice on the DCC page).
Edit (Sep 6): The purpose of the Radd resistors is to lower the resistance and thus up the current through the LED. The equivalent load becomes 287 Ohms. Presumably, this in series with the LED is what gives the 25 mA stated on the schematic. This implies the LED has an effective resistance of 100 Ohms at this operating point. Why 3 resistors? To distribute the heat load. The 1206 SMD resistors are usually rated for 1/4 W. Better to replace with 287 Ohm metal film resistors rated for 1 W, if Steve can find them online.
The ITMX table had relaxed overnight into a slightly misaligned state overnight - since the ITMX table holds PR2 and hence can affect the input pointing, we decided to fix this before commencing alignment work today. The misalignment was not as bad as what Johannes observed prior to his first re-leveling attempt, but was ~1 division on the spirit level. So I decided to move one set of weights to level the table again. It is entirely possible that over the next couple of days, the table will shift slightly again, but the hope is that we are closer to the 'ideal' orientation of the table now... Pictures to follow...
[johannes, lydia, gautam]
GV EDIT Sep 5: These numbers do make sense if the ND filter that was on the Transmon QPD had ND = 0.6 (there are two at the end, one labelled ND 0.6 and the other labelled ND10 though the latter label looks like some custom label so I don't really trust that value), even though only one was on, unfortunately I don't remember which. So, for 10% of input power with a factor of 8 increase because the ND filter is removed and also that the 50% BS has been replaced with a HR mirror, we expect a transmission level of ~0.6 (compared to the normalized value under normal IFO operation) which is close to what we see...
In any case, I think we can work on putting in the X arm now and work on recovering that.
To do for the Y-arm (now that the F.C. is off, we should try and do this in as few chamber openings as possible):
Then we need to do all of this for the X arm as well. The PRM LR coil is still giving no output - I will try moving the bias sliders around to see if this is a stuck magnet situation, but perhaps it is not. Since Eric's 3-satellite-box-monte did not yield any positive results, we have to consider the possibility that the LED or PD themselves are damaged. If so, I don't see any workaround without opening up the BS-PRM chamber, but if we can avoid this, we should. Perhaps when ITMX is open we can use the camera with the IR filter removed to see if all the OSEM LEDs are functional through the beam tube.
We are also piping POY11 error to the DAFI model and can hear it in the control room.
Rana suggested reviving the MC autolocker - I've made some changes to the low power MC autolocker scripts and they've been working the few times I tried today evening, but let's see how it does over the weekend. I've also changed the Y axis of the StripTool on the wall to better reflect the low-power range..
[Gautam, Lydia, Johannes]
The next step is the tip tilt fine alignment of the IR into the arm, using TRY, from which we removed the ND filter for the time being.
I balanced the ITMX and ETMX tables into level position today, for which I had to move quite a few of the on-table weights. I'm recording their original positions for future use here.
This table was only off in 'pitch', I moved the middle weight to a new location as shown in the pictures. I added secondf disk weight on top of the one I moved, this one has to come out again when we install ETMX.
I moved some weights around as shown in the image, but didn't have to add any. We simply have to move them back to their original location when the time comes.
While in the chambers, I also took some pictures of the ETMX window and PR2, motivated by the dirty state of SR2. We might want to consider cleaning both, specifically PR2 is relatively easily accessible and can be cleaned when we open the ITMX chamber to remove its FC and move it back into position.
I'll add a picture of the installation when I get back to campus and finish hooking up the power cable. I haven't added this channel to the actual pages yet because there's not any data right now- the box is still unpowered because my braided power cable wasn't long enough. I just changed the format of the spectrum to ASD and added spectrograms. Here's how the tab looks now: https://ldas-jobs.ligo.caltech.edu/~praful.vasireddy/1155014117-1155015017/pem/acoustic/
Let me know if there's anything else to change.
My box has been suspended in the PSL using surgical tubing, and it has been connected to C1:PEM-MIC_1 (C17) with a BNC. I made a braided power cable as well but it turned out to be slightly too short... Once this is fixed, everything should be ready and we can see if it's working correctly. I also set up a new tab on the summary pages for this channel:
This data is back from when I had my solderless breadboard running near MC2. I'll add this tab to the real pages once the box is working (which could be a while since I'm gone for a month). Let me know if you see any issues with either the tab or the box/cables.
On the bounce roll balancing:
Recall that back in 2006, the main issue was not with the bounce mode coupling into the OSEMs but instead with too much cross-coupling between the damping loops themselves:
Old elogs from Osamu (reader / readonly). Osamu will be here in a couple weeks and can try to explain what he was doing back then.
The problem was that without a good input matrix, the low frequency motion of the suspension point was dominated by the damping noise rather than the seismic noise. The bounce mode is a nice indicator of whether the OSEM is oriented up/down but its not the most important thing. More important is that the magnet is in the actual LED beam, not just the apparent center of the OSEM.
Then we should be able to fix things by running the diagonalization script and correcting the input matrix (which depends somewhat on the DC alignment).
This is an option to isolate the vacuum chamber from the dusty 40m lab: 4x8 HEPA unit or Air Curtain
It does not limit crane operations. Here is some science based approach to air filters
Lets put horizontal and vertical witness plates next to our arm cavity TMs just before pumpdown.
So if the SRM satellite box is good, than the ITMY sensor UL or vacuum cabeling from sersor to sat amp is bad.
Koji tweaked the alignment sliders till we were able to get the Y arm locked to green in a 00 mode, GTRY ~ 0.5 which is the prevent number I have in my head. The green input pointing looks slightly off in yaw, as the spot on the ITM looks a little misaligned - I will fix this tomorrow. But it is encouraging that we can lock to the green, suggests we are not crazily off in alignment.
[Ed by KA: slider values: ETMY (P, Y) = (-3.5459, 0.7050), ITMY (P, Y) = (0.3013, -0.2127)]
While we were locked to the green, ITMY UL coil acted up quite a bit - with a large number of clearly visible excursions. Since the damping was on, this translated to somewhat violent jerking of ITMY (though the green impressively remained locked). We need to fix this. In the interest of diagnosis, I have switched in the SRM satellite box for the ITM one, for overnight observation. It would be good to narrow this down to the electronics. Since SRM is EQ-stopped, I did not plug in any satellite box for SRM. The problem is a difficult one to diagnose, as we can't be sure if the problem is with the LED current driver stage or the PD amplifier stage (or for that matter, the LED/PD themselves), and because the glitches are so intermittent. I will see if any further information can be gleaned in this regard before embarking on some extreme measure like switching out all the 1125 OpAmps or something...
Does anyone know if we have a spare satellite box handy?
Is the spare sat amp is bad ?
Some more numbers we found while working in/around the chamber today:
These numbers were measured using our particle counter, which has a pump rate of 0.1 cfm, so the numbers above are 10x the numbers shown on the instrument after a measurement to account for this.
Essentially, the chamber is pretty dirty. Peeling the F.C with hard to reach optics like the ITM installed in place is not really feasible, and after peeling the F.C, we are looking at a best case of an additional 1-2 weeks in air to align the IFO, during which the optic is apparently exposed to quite a lot of particulates. In fact, with the high intensity flashlight left on, I actually saw some flecks of dust occassionally floating around inside the chamber while I was working on the optic. But this is just something we have to accept I guess.