We've seen for some time now that one of the PRM OSEM signals has been gone, and all of the SRM signals seem dark. We had tried squishing various cables to no avail.
Today I played some "musical satellite boxes," in an attempt to see if the problems are in the chambers or in the signal chains. That is, I swapped the OSEM cables from the vacuum feedthroughs between the satellite boxes, and observed what happened.
It seems clear that something is up with SRM inside the chamber. For PRM, it's not so clear...
Somehow, issues with the LR channel follow both the PRM OSEMs and the PRM satellite box.
PRM LR first went dark on Jul 2nd, after the IFO was vented, but before we took any doors off (which happened on the 5th). I'm not sure what may have caused this.
SRM OSEMS first went dark on the evening of Jul 18, the day before ELOG 12310, when ITMY was moved in the same chamber. Maybe this ELOG was written about work the day before, but the sensors show disturbances over the course of hours. I think we need to double check the connections in chamber.
My box has been suspended in the PSL using surgical tubing, and it has been connected to C1:PEM-MIC_1 (C17) with a BNC. I made a braided power cable as well but it turned out to be slightly too short... Once this is fixed, everything should be ready and we can see if it's working correctly. I also set up a new tab on the summary pages for this channel:
This data is back from when I had my solderless breadboard running near MC2. I'll add this tab to the real pages once the box is working (which could be a while since I'm gone for a month). Let me know if you see any issues with either the tab or the box/cables.
[lydia, johannes, gautam]
While struggling to minimize the bounce mode coupling into the sensor signals, we briefly poked into the ITMY chamber, and think that we understand the origin of the problem, at least for the SRM.
Essentially, we believe that moving the ITM from its nominal position to the edge of the table has shifted the table leveling such that the optic (SRM) is tilted backwards (hence the magnets are completely occluding the LEDs) and that perhaps the optic is in contact with one or more of the bottom EQ stops (hence the signal is stationary, no oscillations visible. The timing of the signals going dark as Eric mentioned supports this hypothesis. The reason why we believe this to be the case is that when I was trying to loosen the screw on the clamp holding the ITMY cage to the table, we saw ~1Hz signals from all 5 SRM OSEM sensors, though they were well away from the nominal equilibrium values. The arrangement of towers in the chamber right now did not permit me to get a good look at the SRM magnets, but I believe they are all still attached to the optic, and that they are NOT stuck to the OSEM coils. If this is indeed the case, putting ITMY back in will solve the issue completely.
It is not clear what has happened to the LR coil on the PRM - could it be that during the venting process, somehow the LR magnet got stuck to the OSEM? If so, can we free it by the usual bias jiggling?
There was some confusion as to the order in which we should go about trying to recover the Y arm. But here are the steps we decided on in the end.
Yesterday, Eric, Johannes and I tried to do step 1, but after some hours of beam walking, we were unsuccessful. Today morning, Koji suggested that the ITM wedge could be playing a part - essentially, over 40m, the wedge would shift the beam horizontally by ~30cm, which is kind of what we were seeing yesterday. That is, with 0 biases to the tip tilts, we could find the beam in the ETM chamber, towards the end of the table, ~30cm away from where it should be (since the input pointing is adjusted taking this effect into account, but we were doing all of our alignment attempts without the ITM in).
So, we shifted strategy today. The idea was to trust that the green beam was well aligned to the cavity axis (we had maximized the green transmission before the vent), and set the pitch bias voltage to ETMY by making the reflected beam overlap with itself. This was done successfully, and we needed to apply a pitch bias of ~-2.70 (value on the MEDM screen slider), which agrees well with what I was seeing in the cleanroom. We then adjusted the OSEMs to bring the sensor outputs to half their nominal maximum value. Next, we went into the ITMX chamber, and were able to find the green beam, at the right height, and approximately where we expect the center of the ITM to be (this supports the hypothesis that the green input pointing was pretty good). I am however concerned if this is truly the right value of the bias for making a cavity with the ITM, because the pre-vent value of the pitch bias slider for ETMY was at -3.7, which is a 30% difference from the current value (and I can't think of a reason why this should have changed, the standoffs weren't touched for ETMY). If we go ahead and fine tune the OSEMs rotationally assuming this is the right bias to have, we may end up with sub-optimal bounce mode coupling into the sensor signals if we have to apply a significantly larger/smaller offset to realise a cavity? The alternative is to put in the ITM, and set the pitch balance using the IR beam, and then go about rotating OSEMs. The obvious downside is that we have to peel the F.C. off, risking dirtying the ITMs.
For much of the rest of the day, we were trying to play with the rotation of the OSEM coils in order to minimize the bounce mode coupling into the sensor signals. We weren't able to come up with a good scheme to do this measurement, and I couldn't find any elog which details how this was done in the past. The problem is we have no target as to how good is good enough, and it is extremely difficult to gauge whether our rotation has improved the situation or not. For instance, with no rotation of the OSEMs, by observing the bounce mode peak height over a period of 20-30 minutes, we saw the peak height change by a factor of at least 3. This is not really surprising I guess, because the impulses that are exciting the bounce mode are stochastic (or at least they should be), and so it is very hard to make an apples to apples comparison as to whether a rotation has improved the situation on.
After some thought, the best I can come up with is the following. If anyone has better ideas or if my idea is flawed, or if this is a huge waste of time, please correct me!
Of course, this method assumes that the excitation into the bounce mode is a constant over time. I'm also attaching the spectrum of the OSEM sensor signals right now - the optic is in the chamber, free swinging (no damping) with the door on (so it is fairly quiet). The LR signal seems to be the best (indeed seems to match the levels in this plot), but it is not clear whether the others can be improved or not.
There was also some concern as to whether we will be able to see the beam in the ETMX chamber once the ITM has been re-installed. Assuming we get 100mW out of the IMC, PRM transmission of 5.5%, and ITM transmission of 1.4%, we get ~35uW incident on the ETM, which while isn't a lot, should be sufficient to see using an IR card.
I've been noticing that the ETMY UL sensor output has been erratic over the last few days. It seems to be jumping around a lot, even though there is no discernable change in any of the other sensor signals. Damping is OFF, which means the sensor signals should just be a reflection of actual test mass motion. But the fact that only one sensor output is erratic leads me to believe that the problem is in the electronics. I've also double checked that we aren't touching any EQ stops. Also, we had centered all the sensor outputs to half their maximum value pretty carefully. But looking at the Striptool traces, I now find that the UL sensor output has settled at some other value. Simply removing the OSEM connector and plugging it in again leads to the sensor output going back to the carefully centered value. Could it be that the photodiode has gone bad? If so, do we have spare OSEMs to use? I will also re-squish the satellite box cables to see if that fixes the problem.
Attachment #1: Sensor output spectra around the bounce mode peak. Nothing was touched inside the chamber between the time this spectrum was taken and the spectrum I put up last night (in fact the chamber was closed)
Attachment #2: UL sensor output is erratic, while the others show no glitching. This supports the hypothesis that the problem is electronic. The glitch itself happened while the chamber was closed.
Attachment #3: The only difference between this trace and Attachment #2 is that the UL connector was removed and plugged in (OSEM wasn't touched)
This problem has existed well before the vent
We do indeed have a box of clean spare OSEMs, it should be out with all of the other boxes of clean stuff we had for the suspension building. You could also try swapping in a different satellite box, to see if the circuit powering the OSEM PD is to blame.
We worked on reducing the bounce mode coupling into the sensor signals today. After some trial and error, essentially following the procedure I had put up in my previous elog, we think we were successful in reducing the coupling. We have now left the optic free swinging, so that we can collect some data and look at a spectrum with finer bandwidth. But as per the methodology we followed, we saw that the peak height corresponding to the bounce mode increased when we rotated the OSEM either side of its current position (except for the side OSEM, which we felt was in a good enough position to warrant not touching it and messing it up - of course only the spectrum will tell us if we are right or not. I also took some pictures with the camera with the IR filter removed, but we couldn't get any real information from these photos. I also checked with Jenne and Jamie who both suggested that they didn't have any metric with which they judged if the rotation of the OSEM was good enough or not. So we will wait to have a look at the spectrum from later tonight, and if it looks reasonable enough, I vote we move on. As Eric suggested, perhaps we can repalce the UL OSEM coil and see if that solves the apparent UL coil problem. Then we should move on to putting the arm cavity together.
Addendum 11pm 26 Aug 2016: I've uploaded the spectra - looks like our tweaking has gained us a factor of ~2 on LL, LR and SD, and no significant improvement on UL and UR compared to yesterdays spectrum.
I wanted to observe the UL coil for any excursions over the weekend. Looking at the 2 day trend, something is definitely wrong. These glitches/excursions are much more pronounced than what is seen in the pre-vent plots Steve had put up.
In order to try and narrow down whether the problem is with the Satellite box or the LED/PD themselves, I switched the Satellite box at the Y end with the Satellite box for ITMY (at ~930pm tonight). Hopefully over a 12 hour observation period, we see something that will allow us to make some conclusion.
It looks like the problem is indeed in the Satellite box. Attachment #1 shows the second trend for the last 12 hours (~930pm 28 Aug 2016 - 930am 29 Aug 2016) for the ITMY and ETMY sensor signals. The satellite boxes for the two were switched during this time (the switch is seen at the leftmost edge of the plots). After the switch, ETMY UL has been well behaved, though ITMY UL shows evidence of excursions similar to what we have been seeing. All the ITMY coils are pulled out of the suspension cage currently, and are just sitting on the optical table, so they should just be reading out a constant value. I think this is conclusive evidence that the problem is with the Satellite box and not the OSEM itself. I will pull the Satellite box out and have a look at its innards to see if I can find the origin of the problem...
I opened up the ETMY satellite box to investigate the glitches seen in the UL sensor output.
Attachments #1 & 2: The connection to J4 from the satellite amplifier goes through a "satellite amplifier termination board", whose function, according to the schematic, is to prevent oscillations of the output amplifiers for the PD outputs. This seems to have been attached to the inside cover of the Satellite box by means of some sort of sponge/adhesive arrangement. The box itself gets rather hot however, and the sponge/adhesive was a gooey mess. I believe it is possible that some pins on the termination board were getting shorted - so if the 100 ohm resistor for the Ul channel that is meant to prevent the output amplifier oscillating was getting shorted, this could explain the problem.
For now, I cleaned off the old sponge/adhesive as best as I could, and used 4 pads of thick double sided tape (with measured resistance > 60Mohm) to affix the termination board to the inside of the box lid. In the ~3 hours since I have plugged the satellite box back in, there has been no evidence of any glitching.
Of course, it could be that the problem has nothing to do with the termination board, and perhaps an OpAmp in the UL signal chain is damaged, but I stopped short of replacing these for now. I plan to push on with putting the IFO back together, and will keep an eye on this problem to see if more action is needed.
Also, if the inside of the ETMY satellite box had this problem of the sponge/adhesive giving way, it may be that something similar is going on in the other boxes as well. This remains to be investigated.
[gautam, johannes, lydia]
We decided to try some different approaches on minimizing the ETMY bounce coupling today, since the peak height in the previously attched spectrum was higher than the previously recorded levels in 2011 for all but the LR OSEM.
How to minimize particles entering the vacuum envelope.
Just the way it was in August 2011 vent and before.
The portable HEPAs were set up at ETMY and ITMY with CP STAT 100 curtains.
The 40m particles on the floor at ITMY 3000-5000 counts of 0.5 micron cf / min and 0.3 micron size particles are 55,000 - 65,000 counts cf / min
At this condition the MET One Counter #3 on the floor inside the tent goes to zero count of 0.5 micron and 20-40 counts cf / min for 0.3 micron when the tent is slightly overpressured.
100 Sapphire prisms ordered. Delivery date 9-30-2016
~60 deg. prisms,
Size A=B=C=2 mm, length 5 mm
Surface quality: 5 micron RMS
Tolerances: +- 0.1 mm
This prism will be used as a mechanical component
No crystal orientation required
There are some issues with 5 mm sapphire prism Atm5. It will cause interference between one of the prisms and the Side OSEM.
Here are some drawings to see the issues with larger wire standoff.
The 2 mm prism will work.with a 1 mm longer dumbell.
Quotes requested from http://photomachining.com/laser-micromachining-photomachining-contact.html and http://www.optocity.com/
Today we installed ITMY into position in the chamber.
We did some quick checks with the green beam and the IR beam. With the help of the custom Iris for the suspension towers, we gauged that both beams are pretty close to the center of the test mass. So we are in a not unreasonable place to start trying to align the beam. Of course we didn't check if the beam makes it to the ETM today.
The SRM OSEM sensor problem seems to have been resolved by moving the ITM back to its place as we suspected. The values are converging, but not to their pre-vent values (attachment #2). We can adjust these if necessary I guess... Or perhaps this fixes itself once the table returns to its neutral position. This remains to be monitored.
In the never-ending B-R mode reduction saga - we found what we think is an acceptable configuration now. Spectrum attached (Attachment #3). The top two OSEMs are now nearly 90 degrees rotated, while the bottom two are nearly horizontal. Anyways I guess we just have to trust the spectra. I should also point out that the spectra change rather significantly from measurement to measurement. But I think this is good enough to push ahead, unless anyone thinks otherwise?
[Lydia, Gautam, Koji, Johannes]
Summary of things done today:
Unless we get lucky and get the green light to flash in the cavity by playing with the mirror alignment, we will open the ETMY chamber tomorrow. On one hand we can look for the reflected green light in the chamber, or alternatively the IR beam transmitted by ITMY. This way we can obtain estimates for the OSEM biasing and perform the final centering of the OSEMs. We will then also address the bounce mode minimization in ITMY and check if the previous orientations still hold.
Koji tweaked the alignment sliders till we were able to get the Y arm locked to green in a 00 mode, GTRY ~ 0.5 which is the prevent number I have in my head. The green input pointing looks slightly off in yaw, as the spot on the ITM looks a little misaligned - I will fix this tomorrow. But it is encouraging that we can lock to the green, suggests we are not crazily off in alignment.
[Ed by KA: slider values: ETMY (P, Y) = (-3.5459, 0.7050), ITMY (P, Y) = (0.3013, -0.2127)]
While we were locked to the green, ITMY UL coil acted up quite a bit - with a large number of clearly visible excursions. Since the damping was on, this translated to somewhat violent jerking of ITMY (though the green impressively remained locked). We need to fix this. In the interest of diagnosis, I have switched in the SRM satellite box for the ITM one, for overnight observation. It would be good to narrow this down to the electronics. Since SRM is EQ-stopped, I did not plug in any satellite box for SRM. The problem is a difficult one to diagnose, as we can't be sure if the problem is with the LED current driver stage or the PD amplifier stage (or for that matter, the LED/PD themselves), and because the glitches are so intermittent. I will see if any further information can be gleaned in this regard before embarking on some extreme measure like switching out all the 1125 OpAmps or something...
Does anyone know if we have a spare satellite box handy?
Some more numbers we found while working in/around the chamber today:
These numbers were measured using our particle counter, which has a pump rate of 0.1 cfm, so the numbers above are 10x the numbers shown on the instrument after a measurement to account for this.
Essentially, the chamber is pretty dirty. Peeling the F.C with hard to reach optics like the ITM installed in place is not really feasible, and after peeling the F.C, we are looking at a best case of an additional 1-2 weeks in air to align the IFO, during which the optic is apparently exposed to quite a lot of particulates. In fact, with the high intensity flashlight left on, I actually saw some flecks of dust occassionally floating around inside the chamber while I was working on the optic. But this is just something we have to accept I guess.
So if the SRM satellite box is good, than the ITMY sensor UL or vacuum cabeling from sersor to sat amp is bad.
Is the spare sat amp is bad ?
This is an option to isolate the vacuum chamber from the dusty 40m lab: 4x8 HEPA unit or Air Curtain
It does not limit crane operations. Here is some science based approach to air filters
Lets put horizontal and vertical witness plates next to our arm cavity TMs just before pumpdown.
On the bounce roll balancing:
Recall that back in 2006, the main issue was not with the bounce mode coupling into the OSEMs but instead with too much cross-coupling between the damping loops themselves:
Old elogs from Osamu (reader / readonly). Osamu will be here in a couple weeks and can try to explain what he was doing back then.
The problem was that without a good input matrix, the low frequency motion of the suspension point was dominated by the damping noise rather than the seismic noise. The bounce mode is a nice indicator of whether the OSEM is oriented up/down but its not the most important thing. More important is that the magnet is in the actual LED beam, not just the apparent center of the OSEM.
Then we should be able to fix things by running the diagonalization script and correcting the input matrix (which depends somewhat on the DC alignment).
I'll add a picture of the installation when I get back to campus and finish hooking up the power cable. I haven't added this channel to the actual pages yet because there's not any data right now- the box is still unpowered because my braided power cable wasn't long enough. I just changed the format of the spectrum to ASD and added spectrograms. Here's how the tab looks now: https://ldas-jobs.ligo.caltech.edu/~praful.vasireddy/1155014117-1155015017/pem/acoustic/
Let me know if there's anything else to change.
I balanced the ITMX and ETMX tables into level position today, for which I had to move quite a few of the on-table weights. I'm recording their original positions for future use here.
This table was only off in 'pitch', I moved the middle weight to a new location as shown in the pictures. I added secondf disk weight on top of the one I moved, this one has to come out again when we install ETMX.
I moved some weights around as shown in the image, but didn't have to add any. We simply have to move them back to their original location when the time comes.
While in the chambers, I also took some pictures of the ETMX window and PR2, motivated by the dirty state of SR2. We might want to consider cleaning both, specifically PR2 is relatively easily accessible and can be cleaned when we open the ITMX chamber to remove its FC and move it back into position.
[Gautam, Lydia, Johannes]
The next step is the tip tilt fine alignment of the IR into the arm, using TRY, from which we removed the ND filter for the time being.
[johannes, lydia, gautam]
GV EDIT Sep 5: These numbers do make sense if the ND filter that was on the Transmon QPD had ND = 0.6 (there are two at the end, one labelled ND 0.6 and the other labelled ND10 though the latter label looks like some custom label so I don't really trust that value), even though only one was on, unfortunately I don't remember which. So, for 10% of input power with a factor of 8 increase because the ND filter is removed and also that the 50% BS has been replaced with a HR mirror, we expect a transmission level of ~0.6 (compared to the normalized value under normal IFO operation) which is close to what we see...
In any case, I think we can work on putting in the X arm now and work on recovering that.
To do for the Y-arm (now that the F.C. is off, we should try and do this in as few chamber openings as possible):
Then we need to do all of this for the X arm as well. The PRM LR coil is still giving no output - I will try moving the bias sliders around to see if this is a stuck magnet situation, but perhaps it is not. Since Eric's 3-satellite-box-monte did not yield any positive results, we have to consider the possibility that the LED or PD themselves are damaged. If so, I don't see any workaround without opening up the BS-PRM chamber, but if we can avoid this, we should. Perhaps when ITMX is open we can use the camera with the IR filter removed to see if all the OSEM LEDs are functional through the beam tube.
We are also piping POY11 error to the DAFI model and can hear it in the control room.
Rana suggested reviving the MC autolocker - I've made some changes to the low power MC autolocker scripts and they've been working the few times I tried today evening, but let's see how it does over the weekend. I've also changed the Y axis of the StripTool on the wall to better reflect the low-power range..
The ITMX table had relaxed overnight into a slightly misaligned state overnight - since the ITMX table holds PR2 and hence can affect the input pointing, we decided to fix this before commencing alignment work today. The misalignment was not as bad as what Johannes observed prior to his first re-leveling attempt, but was ~1 division on the spirit level. So I decided to move one set of weights to level the table again. It is entirely possible that over the next couple of days, the table will shift slightly again, but the hope is that we are closer to the 'ideal' orientation of the table now... Pictures to follow...
In November of 2010, Valera Frolov (LLO), investigated our satellite amplifiers and made some recommendations about how to increase the SNR.
In light of the recent issues, we ought to fix up one of the spares into this state and swap it in for the ITMY's funky box.
The sat amp schematic is (D961289). It has several versions. Our spare is labeled as version D (not a choice on the DCC page).
Edit (Sep 6): The purpose of the Radd resistors is to lower the resistance and thus up the current through the LED. The equivalent load becomes 287 Ohms. Presumably, this in series with the LED is what gives the 25 mA stated on the schematic. This implies the LED has an effective resistance of 100 Ohms at this operating point. Why 3 resistors? To distribute the heat load. The 1206 SMD resistors are usually rated for 1/4 W. Better to replace with 287 Ohm metal film resistors rated for 1 W, if Steve can find them online.
Hi 40m people,
As Rana is saying, the bounce mode does not matter, or we cannot do anything. Generally speaking, the bounce mode cannot be damped by the setting of 40m SUS. Some tweak techniques may damp a bounce mode by res-gain or something, but it is not a proper way, I think.
This is also that Rana is already saying that the important thing is to find a good direction of OSEM to hit the LED beam to the magnet. Even if the magnet is not located at the center of OSEM hole, still you can find the optimal orientation of OSEM to hit the LED beam to the center of magnet by rotating the OSEM.
I know only an old document of T040054 that Shihori summarized how to adjust the matrix at the 40m. Too bad input/output matrix may introduce some troubles, but even roughly adjusted matrix should be still fine.
I will be at Caltech on 12-14 of September. If I can help something, I am willing to work with you!
The attached PDF shows the output noise of the satellite amp. This was calculated using 'osempd.fil' in the 40m/LISO GitLab repo.
The mean voltage output is ~1 Vdc, which corresponds to a current with a shot noise level of 100 nV/rHz on this plot. So the opamp current noise dominates below 1 Hz as long as the OSEM LED output is indeed quantum limited down to 0.1 Hz. Sounds highly implausible.
To convert into meters, we divide by the OSEM conversion factor of ~1.6 V/mm, so the shot noise equivalent would be ~1e-10 m/rHz above 1 Hz.
After adding the sat amp to the 40m DCC tree (D1600348), I notice that not only is the PD readout not built for low noise, neither is the LED drive. The noise should be dominated by the voltage noise of the LT1031 voltage reference. This has a noise of ~500 nV/rHz at 1 Hz. That corresponds to an equivalent current noise through the LED of 25 mA * (500e-9 / 10) ~ 1 nA/rHz. Or ~45 nV/rHz at the sat amp output. This would be OK as long as everything behaves ideally. BUT, we have thick film (i.e. black surface mount) resistors on the LED drive so we'll have to measure it to make sure.
Also, why is the OSEM LED included in the feedback loop of the driver? It means disconnecting the cable from the sat amp makes the driver go unstable probably. I think one concept is that including the device in the feedback loop makes it so that any EMI picked up in the cabling, etc. gets cancelled out by the opamp. But this then requires that we test each driver to make sure it doesn't oscillate when driving the long cable.
If we have some data with one of the optics clamped and the open light hitting the PD, or with the OSEMs removed and sitting on the table, that would be useful for evaluating the end-to-end noise of the OSEM circuit. It seems like we probably have that due to the vent work, so please post the times here if you have them.
The ETMX OSEMs have been attached to its Satellite box and plugged in for the last 10 days or so, with the PD exposed to the unobstructed LED. I pulled the spectrum of one of the sensors (mean detrended, I assume this takes care of removing the DC value?). The DQed channels claim to record um (the raw ADC counts are multiplied by a conversion factor of 0.36). For comparison, re-converted the y-axis for the measured curve to counts, and multiplied the total noise curve from the LISO simulation by a factor of 3267.8cts/V (2^16cts/20V) so the Y axis is noise in units of counts/rtHz. At 1Hz, there is more than an order of magnitude difference between the simulation and the measurement which makes me suspect my y-axis conversion, but I think I've done this correctly. Can such a large discrepancy be solely due to thick film resistors?
I looked at the PRM free swing spectra. The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
Some of the frequencies have changed between the 2011 in-vac measurement and our 2016 in-air measurement, but that seems within usual parameters.
In the morning, Steve will start opening the north BS door so that we can enter to inspect the PRM LR OSEM.
For the ITMY, I squished together the cables which are in the 'Cable Interface Board' which lives in the rack. This thing takes the 64 pin IDC from the satellite module and converts it into 2 D-sub connectors to go to the PD whitening board and the coil driver board. Lets see if the ITMY OSEM glitches change character overnight.
[Teng, Johannes, Lydia, gautam]
The modes look like they're at the right frequencies, so pointing more and more towards a LED or satellite box issue.
We peeked into the BS-PRM chamber via the ITMX chamber to see if we could shed any light on this situation. It's hard to get a picture that is in focus, but it looks quite clear that the LR LED (in the lower left when viewed from the HR side) isn't anywhere near as bright as the rest (see Attachment #1). Various hypothesis include failed LED / piece of Al foil blocking the LED / teflon aperture slipped over the LED. But looks like we can't solve this without opening up the BS-PRM chamber. The plan tomorrow is to open up the chamber, pull out the problematic coil. Once we have a better idea of what is going wrong, we can decide what the appropriate course of action is - replace the OSEM or something else.
As part of the diagnosis, I switched the PRM and SRM satellite boxes earlier today evening around 6pm. They remain in this switched state for now.
Steve, we plan to take the BS-PRM heavy door off tomorrow morning.
Edit 7.30pm: I have managed to recover Y-arm in air locking, and the transmission is up at ~0.6 again which is what we were seeing prior to touching anything on the BS-PRM table, so it looks like the tip-tilt has not gone badly astray... I have also restored the Satellite boxes so that both PRM and SRM have their designated boxes
We would like to establish a system for setting up ADC channels and integrating them into the existing EPICS framework, so that we can gradually switch over channels that are currently handled by the aging slow machines. Otherwise, we will be stuck when they eventually fail. As a preliminary test for this method, we are in the process of setting up an Acromag ADC to read the "Diagnostic" output of the PSL controller. This information will also be useful to monitor the health of the PSL.
Today, we accomplished the following:
The power supply has been turned off for the night.
Detailed elog to follow but summary of todays activities:
Visiting graduate student Teng Zhang from Glasglow received 40m specific safety training yesterday.
[steve, teng, johannes, lydia, gautam]
Depending on how the X arm situation is, we will finish putting back all the heavy doors on Monday and start the pumpdown
GV Edit 11.30pm:
Looks like on Monday, we will look to put the heavy doors on ITMY, ITMX and ETMX chambers, and begin the pumpdown
For the PRM, I aligned it until the arm flashes were maximized and the REFL camera showed a centered spot with dips happening during the arm pops. AS port was more messy since the Michelson alignment wasn't perfect, but the spots were both near the center of the cam and the SRM alignment maximized the wangy fringiness of the image as well as the angry cat meow sounds that the full IFO makes as heard through the DAFI (listening to POX).
On Monday, Osamu should be back and can help with doors and then alignment recovery and locking.
[Gautam, Steve, Johannes]
We put on the remaining heavy doors on the chambers (ITMY, ITMX,ETMX, in this order) this morning. On the ITMY and ETMX tables we placed old OpLev steering mirrors that are clean and baked as witness plates such that may one day provide some insight into dust accumulation on optics.
With the heavy doors on we confirmed that we were still able to lock both IFO arms and used the dither scripts to optimize the alignment. Following that we centered all OpLevs and aligned the X and Y green beams.
We are pumping down. The annuloses are below 10 mTorr
The vacuum gauges are not communicating with the medm so there is no plot available.
The main volume pumping is stopped at P1 = 220 Torr ( New SuperBee 174 Torr ) for overnight.
note: SuperBee was reading 791 Torr at atm and it was not set to 760 !
We ran the scripts to diagonalize the damping matrices using the free swinging data from staurday night/sunday morning. The actual entries used for damping have not been changed. However, we did generate updated matrices for all the main optics (not including the mode cleaner optics, which were not free swinging over the weekend).
The beam splitter that directs light into the MC REFL photodiode has not been replaced; there is still a mirror there. Gautam suggested we wait to replace it until the PSL shutter is open so the beam can be aligned. However, this must be done before going to high power.
GV addendum: What I suggested was to try and recover the arm alignment using the current low power configuration after pumpdown - since we were well aligned just before pumpdown, we should be able to recover this alignment pretty easily at low power. After locking both arms and running the dither align (also center all Oplevs), we can go ahead do the following:
Last night from 8:30 pm to 8:30 am PDT, ETMY UL signal was glitchy again. As of now it seems to have quieted back down, but we pushed on the cables on the board at the Y end to hopefully prevent it from coming back. After doing so it still seems to be behaving well.
P1 IFO pressure is 1 mTorr, valve configuration: vacuum normal, annulosses are pumped, RGA is off, not pumped.
THANKS to Chris !
The shutters can be opened with high power.
No communication error message still exist.
I will reboot c1vac1 and c1vac2 to get gauge communication with medm tomorrow.
The pumpdown had stalled because of some ancient vacuum interlock code that prevented opening the valve V1 between the turbo pump and the main volume.
This interlock  compares the channels C1:Vac-P1_pressure and C1:Vac-PTP1_pressure, neither of which is functioning at the moment. The P1 channel apparently stopped reading sometime during the vent, and contained a value of ~700 torr, while the PTP1 channel contained 0. So the interlock code saw this huge apparent pressure difference and refused to move the valve.
To bypass this check, we used caput to enter a pressure of 0 for P1.
Something strange happened to the ITMX osem reading around 4.pm. PDT as shown below.
Also the there was no response of the reading as we adjusted the PITCH and YAW. :(
Note that we restarted the slow machine: c1susaux,c1ausex this afternoon because of the unresponced interface.
We continued to work on the diagonalization scripts today and devised a way of choosing starting parameters that seems to work much better, and is easier to use, than tuning up to 15 parameters by hand per optic.
We still noticed phase problems with ITMY, which appear to be preventing good diagonalization (See Attachment 1). Almost every degree of freedom has a significant imaginary part in the sensing matrix. We looked at the phases of the cross spectra in DDT and saw that indeed, the OSEM signals do not have the appropriate relative phases at the peak frequencies, especially in PIT and YAW (see Attachment 2: the phase at the peak is about 30 degrees when it should be 180). These phases are different for data takes ~24 hours apart, but are still wrong. We also looked at this information for ETMY and saw the correct behavior. We temporarily moved the pitch and yaw sliders for ITMY and looked at the OSEM response on a striptool, and the signals moved in the expected way. Can anyone suggest a reason why this would be happening? Is there another stretch of data (besides this past weekend) which would be good to compare to?
I believe that the UR and LR magnets are stuck. There was no earth quake at 16:18 yesterday. Something had to kick it into this position. See 4days plot
Please advise freeing details.