With all the PCIe issues now resolved, yesterday I proceeded to build an IOP model for each of new FEs. I assigned them names and DCUIDs consist with the 40m convention, listed below. These models currently exist on only the cloned copy of /opt/rtcds running on the test stand. They will be copied to the main network disk later, once the new systems are fully tested.
The models compile and install successfully. The RCG runtime diagnostics indicate that all is working except for the timing synchronization and DAQD data transmission. This is as expected because neither of these have been set up yet.
The next step is to provide the 65 kHz clock signals from the timing fanout via LC optical fiber. I overlooked the fact that an SPX optical transceiver is required to interface the fiber to the timing slave board. These were not provided with the timing slaves we received. The timing slaves require a particular type of transceiver, 100base-FX/OC-3, which we did not have on hand. (For future reference, there is a handy list of compatible transceivers in E080541, p. 14.) I placed a Digikey order for two Finisar FTLF1217P2BTL, which should arrive within two days.
Today I brought and installed the new optical transceivers (Finisar FTLF1217P2BTL) for the two timing slaves. The timing slaves appear to phase-lock to the clocking signal from the master fanout. A few seconds after each timing slave is powered on, its status LED begins steadily blinking at 1 Hz, just as in the existing 40m systems.
However, some other timing issue remains unresolved. When the IOP model is started (on either FE), the DACKILL watchdog appears to start in a tripped state. Then after a few minutes of running, the TIM and ADC indicators go down as well. This makes me suspect the sample clocks are not really phase-locked. However, the models do start up with no error messages. Will continue to debug...
Did you match the local PC time with the GPS time?
We received the following I/O chassis parts from LLO:
We picked up the following I/O chassis components from Downs CDS testing lab:
The MC_ IDC 64 pin cables from sat. amplifiers to junction-interface-board towards whitening - dewhitening at the back of rack 1 X 5 are finally clamped with
All other sus cables of the same kind have the correct short latch arm to lock them in for reliable contact.
Arms and PRC hold well, powers are within normal ranges. (TR[X,Y] >.9, POP110I > 400)
Then, Jenne took over, worked some alignment magic, and did the hard part of getting the Yarm locked and ASS'd.
Green no longer locks to 00 on the Y-arm, and the X-arm green transmission isn't the best despite PZT fiddling (~.6). Also, when green is locked to the Xarm, I see a distinct circular spot on the GTRY camera, with Y green and PSL green shutters closed.
While Jenne used Yarm ASS successfully, when I run it now, it slowly pushes things out of alignment, and two of the traces (YARM_ETM_YAW_L_DEMOD_I_OUTPUT, and the same for ITM) have a reasonable constant offset that doesn't move away.
The Yarm ASS is now working (as is the Xarm ASS). Both of the TT's pitch servos had a sign flip. We don't know why.
To start, we lowered the matrix elements that push on the TTs by a factor of 3, to compensate for the new factor of 3 in the slider gains: ezcastep C1:ASS-YARM_OUT_MTRX_5_5 /3 C1:ASS-YARM_OUT_MTRX_5_7 /3 C1:ASS-YARM_OUT_MTRX_6_6 /3 C1:ASS-YARM_OUT_MTRX_6_8 /3 C1:ASS-YARM_OUT_MTRX_7_5 /3 C1:ASS-YARM_OUT_MTRX_7_7 /3 C1:ASS-YARM_OUT_MTRX_8_6 /3 C1:ASS-YARM_OUT_MTRX_8_8 /3
ezcastep C1:ASS-YARM_OUT_MTRX_5_5 /3 C1:ASS-YARM_OUT_MTRX_5_7 /3 C1:ASS-YARM_OUT_MTRX_6_6 /3 C1:ASS-YARM_OUT_MTRX_6_8 /3 C1:ASS-YARM_OUT_MTRX_7_5 /3 C1:ASS-YARM_OUT_MTRX_7_7 /3 C1:ASS-YARM_OUT_MTRX_8_6 /3 C1:ASS-YARM_OUT_MTRX_8_8 /3
We turned off all 4 tip tilt ASS servos (in the Yarm ASS servo screen), and turned them on one at a time. By doing this, we discovered that the pitch servos for both TT1 and TT2 needed to have the opposite sign from what they used to have. However, the yaw servos kept the original signs. It really doesn't make sense to me why this should be, but this is the way the ASS servo works. We left both Xarm and Yarm ASSs on for several minutes, and saw that they didn't push any mirrors out of alignment.
The ASS_DOTHER_ON burt snapshot has been resaved with the new values.
Also, earlier this evening, I aligned the Yarm green beam to the cavity, although the cavity was not optimally aligned, so this needs to be re-done.
On our to-do list should be to add the tip tilt slider values to the DAQ channels list.
After the mini boot fest that Jenne did today, I checked whether that fixed the overflow issues we yesterday prevented the alignemnt of the arms.
I ran the alignment script for the arms getting 0.85 for TRX and 0.75 for TRY: low values.
After I ran the script ,C1SUSVME1 and C1SUSVME2 started having problems with the FE SYNC (counter at 16378). I rebooted those two and fix the sync problem but the transmitted powers didn't improve.
Are we still having problem due to MC misalignment?
Tonight I aligned the IFO by running the scripts one by one.
SRC was far off and I had to align SRM by hand before the script could work. SPOB is still low when DRM is aligned.
I'm restoring the full IFO now that I'm taking off.
There was no progress tonight after Jenne left.
I could not find any reasonable fringes of the IFO after 3 hours of optics jiggling.
* I jiggled TT1 and TT2. The slider has not been restored.
We should probably look at the value in the day time and revert them.
(Still this does not ensure the recovery of the previous pointing because of the hysteresis)
* The arms are still aligned for the green.
It's not TEM00 any more because of the vent/drift but the fringe is visible (i.e. eigenaxis is on the mirror)
* As we touched PR3, the input pointing is totally misaligned.
To Do / Plan
* We need to find the resonance of the yarm by the input TTs. Once the resonance is found, we will align the PRM.
* Move the BS to find the xarm resonance.
* Finally align SRM
* It was not possible to find the resonance of the yarm without going into the chamber. Definitely we can find the spot on the ITMY by a card, but we are not sure the beam can hit the ETMY. And the baffles makes the work difficult.
* One possibility is to align the input beam so that the ITMY beam is retroreflected to the PRM. I tried it but the beam was not visible form the camera.
I have compressed the IFO Configure screen. All PRMI things (sideband lock and carrier lock) are in the PRMI button, all arm things (both RF and ALS) are in the respective arm buttons.
I have also made a new set of scripts for CARM and DARM lock acquisition with ALS.
I hope that each button's purpose is clear, but take a second to look at them before you next use the IFO Configure screen.
The Maglev is running for 10 days with V1 closed. The pressure at the RGA-region is at 2e-9 torr on CC4 cold cathode gauge.
Valve VM2 to Rga-only was opened 6 days ago. The foreline pressure is still 2.2e-6 torr with small Varian turbo ~10 l/s on cc2
Daily scans show small improvement in large amu 32 Oxygen and large amu 16, 17 and 18 H20 water peaks.
Argon calibration valve is leaking on our Ar cylinder and it is constant.
The good news is that there are no fragmented hydrocarbons in the spectrum.
The Maglev is soaked with water. It was seating in the 40m for 4 years with viton o-ring seals
However I can not explan the large oxygen peak, either Rai Weiss can not.
The Maglev scans are indicating cleanliness and water. I'm ready to open V1 to the IFO
V1 valve is open to IFO now. V1 interlock will be tested tomorrow.
Valve configuration: VAC NORMAL with CRYO and Maglev are both pumping on the IFO
The IFO RGA scan is normal.
The Cryo needs to be regenerated next. It has been pumping for 36 days since last regenerated.
This has to be done periodically, so the Cryo's 14 K cold head is not insulated by by ice of all things pumped away from the IFO
When I came in this morning:
Checking status of slow machines, it looked like c1sus, c1aux, and c1iscaux needed reboots, which I did. Still PMC would not lock. So I did a burtrestore, and then PMC was locked. But there seemed to be waaaaay to much motion of MCREFL, so I checked the suspension. The shadow sensor EPICS channels are reporting ~10,000 cts, while they used to be ~1000cts. No unusual red flags on the CDS side. Everything looked nominal when I briefly came in at 6:30pm PT yesterday, not sure if anything was done with the IFO last night.
Pending further investigation, I'm leaving all watchdogs shutdown and the PSL shutter closed.
A quick look at the Sorensens in 1X6 revealed that the +/- 20V DC power supplies were current overloaded (see Attachment #1). So I set those two units to zero until we figure out what's going on. Possibly something is shorted inside the ITMX satellite box and a fuse is blown somewhere. I'll look into it more once Steve is back.
The C1:IFO-STATE variable is actually a bunch (16 to be precise) of bits, and the byte they form (2 bytes) converted to decimal is what is written to the EPICS channel. It was reported on the call today that the nominal value of the variable when the IMC is locked was "8", while it has become "10" today. In fact, this has nothing to do with the IMC. You can see that the "PMC locked" bit is set in Attachment #1. This is done in the AutoLock.sh PMC autolocker script, which was run a few days ago. Nominally, I just lock the PMC by moving some sliders, and I neglect to set/unset this bit.
Basically, there is no anomalous behavior. This is not to say that the situation cannot be improved. Indeed, we should get rid of the obsolete states (e.g. FSS Locked, MZ locked), and add some other states like "PRMI locked". While there is nothing wrong with setting these bits at the end of execution of some script, a better way would be to configure the EPICS record to automatically set / unset itself based on some diagnostic channels. For example, the "PMC locked" bit should be set if (i) the PMC REFL is < 0.1 AND (ii) PMC TRANS is >0.65 (the exact thresholds are up for debate). Then we are truly recording the state of the IFO and not relying on some script to write to the bit (I haven't thoguht through if there are some edge cases where we need an unreasonable number of diagnostic channels to determine if we are in a certain state or not).
That makes sense. I assumed that IFO-STATE is configured as you have proposed it to be configured. This could be implemented in later.
a better way would be to configure the EPICS record to automatically set / unset itself based on some diagnostic channels. For example, the "PMC locked" bit should be set if (i) the PMC REFL is < 0.1 AND (ii) PMC TRANS is >0.65 (the exact thresholds are up for debate). Then we are truly recording the state of the IFO and not relying on some script to write to the bit (I haven't thoguht through if there are some edge cases where we need an unreasonable number of diagnostic channels to determine if we are in a certain state or not).
[Yuta, Manasa, Jenne, Jamie, Steve]
0. Measured MC centering (off by 5mrad) before getting the doors off.
1. Got the TTs to 0.0 in pitch and yaw.
2. Using the MMTs, the beam was centered on the TTs.
3. TT1 was adjusted such that the incident beam was centered at PRM (with target).
4. TT2 was adjusted such that the beam passed through the center of BS (with target).
5. Centered the beam on PR2 by sliding it on the table.
6. Moved PR2 and tweaked TT2 to center the beam on ITMY and BS respectively.
7. Using TTs, we got the beam centered on ETMY while still checking the centering on ITMY.
8. ITMY was adjusted such that it retro-reflected at the BS.
9. ETMY was aligned to get a few bounces in the arm cavity.
10. Centered on ITMX by adjusting BS and then tweaked ITMX such that we retro-reflected at BS.
11. At this point we were able to see the MI fringes at the AS port.
12. Tweaked ITMX to obtain reflected MI fringes in front of MMT2.
13. By fine adjustments of the ITMs, we were able to get the reflected MI to go through the faraday while still checking that we were retro-reflecting at the BS.
14. Tweaked the PRM, such that the PRM reflected beam which was already visible on the 'front face back face of faraday' camera went through the faraday and made fine adjustments to see it fringing with the reflected MI that was already aligned.
15. At this point we saw the REFL (flashing PRMI) coming out of vacuum unclipped and on the camera.
16. Started with alignment to get the AS beam out of vacuum. We tweaked OM1 and OM2 (steering mirrors in the ITMY chamber) to center the beams on OM4 and OM3 (steering mirrors in the BSC) respectively.
17. We then adjusted steering mirrors OM5 and OM6 (in the OMC chamber) such that the beam went unclipped out of vacuum.
18. Note that we took out the last steering mirror (on the AS table) in front of the AS camera, so that we can find the AS beam easily. This can be fixed after we pump down.
0. REFL still looks like an egg, but leave it .
1. Align PRMI (no more in-vac!) .
2. Align POP/REFL/AS cameras and PDs.
3. Setup PRM/BS/ITMX/ITMY oplevs.
4. Balance the coils on these mirrors.
5. Lock PRMI.
Yuta and Manasa, you guys are awesome!
Small, inconsequential point: The camera image in the upper right of your video is the *back* of the Faraday in our usual nomenclature. The camera is listed in the videoswitch script as "FI_BACK". The camera looking at the "front" of the Faraday is just called "FI".
After the meeting, I aligned the IFO to the IR, and then I aligned the Ygreen to the Yarm. I then found the beatnotes and used ALS to hold the arms with CARM/DARM, locked the PRMI, and reduced the CARM offset until I had arm powers of about 3. Given that this was at 3pm, and people were tromping all over inside the IFO room, I feel positive about tonight.
So, IFO seems ready, carm_cm_up script was successful, and got me to arm powers of 1, and then I further reduced the offset by a bit to go a little higher.
[Keiko, Jamie, Kiwamu, Anamaria,
We followed the procedure that we laid-out in our elog of yesterday. We completed the first six steps and we now have the y-arm well aligned to the green beam which passes through the center of of both ETMY and ITMY.
The IR beam was steered with the PZTs to coincide with the green beam. The BS was adjusted to see IR beam scatter on a target placed near the center of the ETMX. And then the AS IR beam was steered to the AS camera by adjusting several components along OM path ( we touched OM1, OM2, OM3, OM4, OM5, OMPO and OM6). We then looked for IR fringes in the AS port from the Y-arm. But no luck there. We need to realign the IR beam into the Y-arm cavity axis using the pzts.
We aligned ITMX and PRM to get power recycled Michelson fringes at the AS.
Yuta, Manasa, Jamie, Jenne, Steve, Rana
Starting this morning, we removed the temporary half PRC mirror in front of BS and started to align the IFO in prep for an in-air lock of the PRMI.
This morning, using the new awesome steerable active input TTs, Jenne and I centred the beam on PRM, PR2/3, BS, ITMY and ETMY.
After lunch, Yuta and Manasa aligned the Y ARM, by looking at the multi-pass beam. The X-end door was still on, so they roughly aligned to the X ARM by centring on ITMX with BS. They then got fringes at the BS, and tweaked the ITMs and PRM to get full fringes at BS.
We're currently stuck because the REFL beam appears to be clipped coming out of the faraday, even though the retro-reflected beam from PRM is cleanly going through the faraday output aperture. The best guess at the moment is that the beam is leaving MC at an angle, so the retro-reflected beam is coming out of the faraday at an angle. We did not center spots on MC mirrors before we started the alignment procedure today. That was dumb.
We may be ok to do our PRMI characterization with the clipped REFL, though, then we can fix everything right before we close up. We're going to need to go back to touch up alignment before we close up anyway (we need to get PR2 centered).
Yuta and Manasa are finishing up now by making sure the AS and REFL beams are cleanly existing onto the AS table.
Tomorrow we will set up the PRM oplev, and start to look at the in-air PRMI. Hopefully we can be ready to close up by the end of the week.
We should check MC spot positions to see what they are.
Also, I'm not thrilled about the idea of a clipped REFL beam. Haven't we played that game before, and decided it's a crappy game? Can we recenter the MC, and recover quickly with TT1?
Lot's of alignment work, still no AS beam. REFL is clipped by Faraday output aperture......
Our guess is that this is because we skipped MC centering.
Alignment procedure we took:
1. AM work: Aligned input beam using TT1/TT2
such that the beam hits ETMY and ITMY at the center.
2. Coarsely aligned ITMY
such that the ITMY retro-reflected beam hits BS at the center.
3. Aligned ETMY (we didn't actually move ITMY)
such that Y arm flashes.
This tells you that ITMY is aligned well to the incident beam.
4. Aligned BS
such that the beam hits ITMX at the center.
5. Aligned ITMX
such that the ITMX retro-reflected beam hits BS at the center.
At this point, we saw MI fringes at AS port.
6. Fine alignment of ITMX:
MI reflected beam was not overlapping in front of BS after it was reflected by PRM.
We used this longer REFL path to tune alignment of ITMX to ITMY reflected beam.
We saw MI fringe at REFL port coming out of the chamber, but it was clipped.
7. Aligned PRM
by looking at REFL beam from PRM on the back face of Faraday (video FI_BACK).
We fine tuned the alignment such that PRM retro-relfected beam hits BS at the center and REFL beam from PRM overlaps with the MI fringes at the back face of Faraday.
8. Clipping of REFL at the Faraday output aperture:
We confirmed that the shape of the REFL beam from PRM was OK at the back face of Faraday. But some how, it was clipped at the output aperture. So, REFL beam coming out of the chamber is clipped now.
9. Tried to get AS beam out of the chamber:
We tweaked steering mirrors after SRM to get AS beam out of the chamber. But, we lost the AS beam between the very last folding mirrors (OMPO and OM6) in the OMC chamber......
1. Why clipping at the Faraday output aperture?
In principle, if PRM reflects the incident beam at normal incidence, it should pass the Faraday unclipped. But it's not!
Our guess is that the incident beam does not go well centered through the apertures of the Faraday. I think we have to do MC centering to get good pointing to the Faraday.
We also see that MI fringe at the back face of the Faraday is at the edge of its aperture, after all of these alignment work (we even used Y arm!). This tells you that some thing is wrong.
2. Why did you guys lose the AS beam?
AS beam is too weak after reflecting off of OMPO. The beam was neither visible on IR cards nor IR viewers. The beam is weaker than usual because PMC transmission is ~0.7 and MC REFL is getting high (~ 0.7). We didn't want to realign MC after all of this work today.
Tomorrow (my suggestion):
1. Align PMC (for higher power).
2. MC centering.
3. Input beam steering using TTs and redo the same alignment procedure (it shouldn't take longer than today).
==> Center beam on PR2 (Added by Manasa)
4. Maybe we should better check PRM reflection at REFL port after the Faraday, before doing the full alignment work.
5. Align AS, REFL, POP PDs/cameras.
6. Setup PRM/BS/ITMX/ITMY oplevs.
7. Balance the coils on these mirrors.
8. Lock PRMI.
What needs to be done before pumping down:
1. PRMI characterization: PR gain and g-factor
How can we do the g-factor measurement? Use additional laser? Kakeru method (elog #1434; we need to calibrate mirror tilt to do this)?
2. Glitch study in PRMI locking. If still glitchy, we have to do something. How is beam spot motion? (elog #6953)
3. Fine alignment of the flipped PR2.
4. Fine alignment of IFO using both arms.
The alignment of the interferometer goes basically step by step.
Tuesday will be an alignment day.
0. MC beam centering (it's done)
1. F2P to balance the coils on every optics including BS, PRM, SRM, ITMs and ETMs (Kiwamu).
2. A2L and then change the DC bias of ITMY and ETMY to get a perfect eigen axis (VF/Jamie).
3. align input PZT mirrors (PZT1 and 2) to maximize the Y arm transmission (VF/Jamie).
4. do the same things for X arm but using BS instead of the PZTs.
5. Alignment of the central part.
6. Make a script to automatically get those things done.
I noticed today, and Rana said that he saw Saturday, that the MC refl value when the MC is unlocked is unusually high. It typically goes to about 4.5 V, but now is going up to 6.5V. Since the PMC output is the same as usual (max seen has been about 0.82 today), something must have happened between the PMC and the IMC.
Late last week, EricG and Nichin were looking at things on the AS table. Was anything touched on the AS table? Was anything touched on the PSL table? 'Fess up please, so that we can pinpoint what the change was.
Also, this afternoon, I touched up the MC alignment a bit, although it still needs work (I've asked Manasa to look at it tomorrow). Rana centered the WFS to my MC alignment (this will need to be redone after the MC is truely aligned), and we turned the WFS on. I also locked both arms individually, and locked MICH and PRMI sideband. The PRMI wasn't especially stable unless I turned on the POP ASC. I assume (hope) that this is just because I was doing it during the day, and not because there is something actually different about the PRMI since the computer meltdown.
Rana and I also took some notes on things that need to be done, starting tomorrow (the first line and the yellow line are scribbles):
1. Recovered MC alignment and locked it manually after the ottavia cron failed to help.
2. Measured the MC spots and could not get the MC spots better looking than this.
spot positions in mm (MC1,2,3 pit MC1,2,3 yaw):
[1.6609930611758554, -1.4655939584833202, 1.3133188677545831, -1.9483764375494121, -1.6539541350121174, -0.8862744551298497]
3. Realigned the beams to the MC WFS and enabled WFS servo.
MC Trans SUM is ~17000 counts and MC REFL is ~0.5 counts.
IOO QPDS center
Recover REFL 33
MC autolocker cron
PRMI locking with REFL33 is fine. As it was yesterday, it's a little wobbly without the ASC (just PRM oplev), but I don't know that it's any different than it used to be. It'll hold for long periods of time, so I feel okay about it.
When the PRMI is locked, you can push the "up" button on the ASC screen, and it'll turn down the PRM oplev gain by a large factor, and engage the ASC. When you lose lock, press the "down" button to undo these changes. (Probably the ifoconfig script should include the ASC down script). These up and down scripts for the ASC are already included in the carm_up script (the ASC up), and the watch scripts which run a down script (including ASC down) for the whole IFO when ALS loses lock. If the ASC is engaged, I get bored of watching it before the PRMi loses lock on its own, so I think it's okay. (Let's say that means I've watched it stay locked for at least a few tens of seconds, but it looks like it always has with the ASC - like it'll stay forever).
The only thing that seems different about the PRMI is that I've increased the PRCL gain from -0.02 to -0.04. This is a value that it was at some weeks ago, and then we turned it down for loop osc reasons, but now it doesn't want to catch lock with the lower value, and if I turn it down after it's locked, it has trouble holding on. I have included this change into the PRMI sideband configure script.
I haven't tried anything creative like locking with REFL 165. I also didn't lock with 11 or 55, since 33 just worked.
Nope, we did not touch any of the PDs other than AS55. I have mentioned in my elog:10037 what we did exactly.
We just looked at all the other PDs to check if they were being illuminated by the correctly labeled fiber. Nothing other than that.
All optics have been re-aligned. Jon/Johannes will elog about the work today.
[Jenne, Q, Diego]
OMG, today sucked alignment-wise. Like, wow.
I think that the problem with the ASS is with the input pointing part of the system. I found that if I disable the TTs for the Yarm (iin practice, the outputs are held at zero), I could run the Yarm ASS at full gain of 1, and it would do an awesome job. The first time I did this, I by-hand optimized the TTs by running the test mass loops to make them follow the input pointing. After that, I haven't touched the TT pointing at all, and we've just been running the test mass loops for the Yarm ASS. The Xarm seems to not have this problem (or at least not as drastically), so I left it as it was, touching BS as well as ITMX and ITMY, although the gain still needs to be about 0.3.
I feel pretty good about the IFO alignment now, although it is slightly different than it has been. The transmitted arm powers are higher than they were before I changed the ASS procedure, and there seems to be a little less power fluctuation with alignment. Q points out that I don't have concrete evidence that this is a good alignment, but it feels right.
It was a significant enough change that I had to go down to the Yend to realign the green to the new arm axis. Xgreen we did with the remote PZTs. I also realigned both of the beatnotes on the PSL table.
While I was on the PSL table, I quickly touched up the PMC alignment.
The biggest problem, the one that sucked up more hours and energy than I'd like to admit, is ETMX's jumping. So frustrating. Sometimes it is time-coincident with engaging the LSC, sometimes not. I thought that it might be because there are too many violin filters, but even if I turn off all violin filters to ETMX, it jumped a few times while the cavity was locked. Sometimes it moves when the cavity is just locked and seems happy, sometimes it moves when nothing is resonating except for the green. It takes a few minutes to recover the alignment enough to lock, and then it'll jump again a few minutes later. I haven't gone down to squish the cables today, although I did it yesterday and that didn't seem to do anything.
We had a few hours of time when it wasn't jumping, so we tried a few times to lock the IFO. The last several times we have lost lock because the PRC loop rang up. We measured the loop at low-ish arm powers, but it kept losing lock at higher powers before we could measure. At least 3 times, the PRC lockloss took out CARM and DARM too.
Anyhow, it has been a long day of not accomplishing anything interesting, but hopefully the IFO will feel better tomorrow.
Information acknowledged from Steve:
The last steering mirror mount for IR on the PSL was swapped for a more robust one. Prior to swapping the ibeam positions on the PSL IOO QPDS in ang and pos were recorded.
What I did henceforth:
1. Once the last steering mirror was installed, I walked the beam to restore input pointing using the last 2 steering mirrors. It needed a lot of work in yaw as expected.
2. When the input pointing was recovered, MC locked right away in TEM00. I measured the MC spot positions and compared it with Jenne's measurement made prior to the swap. The spot positions were pretty close.
3. The input pointing was adjusted in pitch and yaw (on the last steering mirror) in small steps. MC alignment was recovered and spot positions were measured each time. After several iterations, the MC spot positions were pretty much centered. I recentered the WFS and reset the WFS offsets. MC is now locked with WFS enabled at ~16900 counts.
4. Since the arms were aligned this morning, I used the Y arm as reference and corrected for the input pointing using tip-tilts.
5. Arms locked right away. Note: ASS doesn't seem to be doing it's job. I had to manually align the arms for maximum on TRX and TRY.
6. MICH and PRMI lock were also recovered.
7. I started to check the status with ALS as well. But for reasons unknown, I don't see any ADC counts corresponding to the beat note. Looking at the beatbox there aren't any signs of disconnected cables. I am saving this as a morning job to fix it.
[EricQ, Gabriele, Manasa]
We found we had lost the Y arm pointing from yesterday. We tried to recover the pointing for a couple of hours and finally decided to take the ETMY heavy door off.
The input beam was aligned to the Y arm. We also got AS and REFL out of vacuum and on the cameras.
We put back the light doors and tried to lock the arms, but did not succeed as yet.
Things to do:
1. Lock arms for IR
2. Realign POP path
3. Recenter all oplevs
4. Try to check the state of PRC after the length change
5. Take in-vacuum pictures
We are close to the end of the vent except for a couple of issues.
* POP is not visible on the IR card. But we see POP flashes unclipped on the camera and also spikes in POP DC. So we are assuming that the POP path hasn't gone far off. If anybody has suggestions for a better method to check this, we could give it a try.
* PRM suspension has not been behaving well. PRM is being kicked around every 5-10 seconds when the PRC is aligned (as seen on REFL camera). We are not sure where this is coming from. The first time we saw this happening was when we were trying to lock PRC at low power even before we took the heavy doors off. So we are pretty sure this is not caused by the foil cover on the OSEMs. We tried turning ON/OFF the oplev servo, turning ON/OFF the damping loops and also checked the connections in the feedthrough and satellite box for the PRM. The OSEM sensor values for the suspension also seem to match the ones on the wiki.
GET CAMERA IMAGES OF EVERYTHING
Check all OpLevs centered, in and out of vacuum
Close PSL shutter & green shutters at the ends
I finished measuring the AbsL for this morning. The IFO is again available.
Please don't mess up with the interferometer though. I'll be back in a couple of ours
The problem with the ASS turned out to be a mode that was rung up at 1326Hz in ETMX. It was rung up when the Xarm's overall gain was too high. So, by turning down the digital gain we were able to prevent it ringing up, and then the ASS worked. To circumvent this, we added a notch to the violin filter bank. It turned out that, upon trying to check if this existed also for the Yarm by turning up the digital gain, the ETMY frequency was almost identical. So, the same single notch is in both ETMs, and it covers the modes for both ETMs.
After that, we got back to locking. We have made at least 9 transitions to all-RF (both CARM and DARM) tonight (I have lost track of how many Den has done while I've been writing this - maybe we're up to 10 or so.). We have changed the order of things a little bit, but they're mostly similar to last week. There are some new notches in the CARM_B filter bank, as well as a 700Hz low pass. We have not been using the lead filter in DARM from last week. Script is checked in, and also zipped and attached. At first CARM was actuating on ETMs, but the last half of the locks we've been using MC2. The script is optimized for MC2 actuation.
While locked all RF, we phased REFL55 in preparation for transitioning PRMI over from REFL165. REFL55 phase was +125, now is +80, give or take 5 deg. We have tried measuring the relative gain and sign between REFL55 and REFL165, but we keep losing lock, perhaps as a result of the TFs Den is taking. He's being gentle though.
Measure CARM loop (why was SRmeasure not working?? is it plugged in??)
Turn on AO boosts, etc.
Earlier today, I rebooted a few unresponsive VME crates (susaux, auxey).
The IMC has been unhappy for a couple of days - the glitches in the MC suspensions are more frequent. I reset the dark offsets, minimized MCREFL by hand, and then re-centered the beam on the MC2 Trans QPD. In this config, the IMC has been relatively stable today, although judging by the control room StripTool WFS control signal traces, the suspension glitches are still happening. Since we have to fix the attenuator issue anyways soon, we can do a touch-up on IMC WFS.
I removed the DC PD used for loss measurements. I found that the AS beam path was disturbed - there is a need to change the alignment, this just makes it more work to get back to IFO locking as I have to check alignment onto the AS55 and AS110 PDs.
Single arm locking worked with minimal effort - although the X arm dither alignment doesn't do the intended job of maximizing the transmission. Needs a checkup.
PRMI locking (carrier resonant) was also pretty easy. Stability of the lock is good, locks hold for ~20 minutes at a time and only broke because I was mucking around. However, when the carrier is resonant, I notice a smeared scatter pattern on the ITMX camera that I don't remember from before. I wonder if the FF idea can be tested in the simpler PRMI config.
After recovering these two simpler IFO configurations, I improved the cavity alignment by hand and with the ASS servos that work. Then I re-centered all the Oplev beams onto their respective QPDs and saved the alignment offsets. I briefly attemped DRMI locking, but had little success, I'm going to try a little later in the evening, so I'm leaving the IFO with the DRMI flashing about, LSC mode off.
I edited the configure scripts (those called from the C1IFO_CONFIGURE screen) for restore XARM and YARM. These used to misalign the ITM of the unused arm, which is totally unnecessary here, as we have both POX and POY. They also used to turn off the drive to the unused ETM. I've commented out these lines, so now running the two restores in series will leave a state where both arms can be locked. This also means that the ITMs will never be deliberately mis-aligned by the restore scripts.
I have modified the IFOconfigure scripts and the corresponding .req files for the X arm and Y arm in burt. I have also added configure scripts to save and restore LSC settings for locking the arms using ALS error signals.
The settings are yet to be saved and the scripts should also be checked if they are working as required.
The steps from this elog were followed.
In addition, I did a burt restore of c1sus, c1mcs.
I then swapped all the gain settings from ITMX to ITMY, and reenabled the watchdogs.
I did some basic kick tests (1000 counts into UL coil) and confirmed channels like C1:SUS-ITMX_ULPD_VAR (watchdogs mV readback) corresponded to the correct optic. I also checked that the POS, PIT, YAW, SIDE produced reasonable damping when engaged.
Overnight, the pressure of the main volume only rose by 10 mtorr, so there was no need to run the roughing pumps again. So we went straight to the turbos - hooked up the AUX drypump and set it up to back TP2. Initially, we tried having both TP2 and TP3 act as backing pumps for TP1, but the wimpy TP3 current was always passing the interlock threshold. So we decided to pump down with TP3 valved off, only TP2 backing TP1. This went smooth - we had to keep an eye on P2, to make sure it stayed below 1 torr. It took ~ 1 hour to go from 500 mtorr to 100 mtorr, but after that, I could almost immediately open up RV2 completely. A safe setting to run at seems to be to have RV2 open by between 0.5 and 1 turn (out of the full range of 7 turns) until the pressure drops to ~100 mtorr. Then we can crank it open. We are, at the time of writing, at ~8e-5 torr and the pressure is coming down steadily.
I had to manually clear the IG error on the CC1 gauge, and re-enabled the High Voltage, so that we have a readback of the main volume pressure in that range. I made a script to do this (enable the HV, the IG error still has to be cleared by pushing the appropriate buttons on the Hornet), it lives at /opt/target/python/serial/turnHornetON.py. I guess it'll take a few days to hit 8e-6 torr, but I don't see any reason to not leave the turbos running over the weekend.
Remaining tasks are (i) disconnect the roughing pump line and (ii) pump down the annuli, which will be done later today. Both were done at ~2pm, now we are in the vacuum normal config. I'll turn the two small turbos to run on "Standby Mode" before I head home today. I think TP3 may be close to end-of-life - the TP3 current went up to 1A even while evacuating the small volume of the annular line (which was already at 1 torr) with the AUX drypump backing it. The interlock condition is set to trip at 1.2A, and this pump is nominally supposed to be able to back TP1 during the pumpdown of the main volume from 500 mtorr, which it wasn't able to do.
It's been an iffy last few hours here at the 40m. Kiwamu, Koji and I were all sitting at our desks, and the computers / RFM network decided to crash. We brought all of the computers back, but now the RefCav and PMC don't want to lock. I'm a wee bit confused by this. Both Kiwamu and I have given it a shot, and we can each get the ref cav to sit and flash, but we can't catch it. Also, when I bring the PMC slider rail to rail, we see no change in the PMC refl camera. Since c1psl had been finicky coming back the first time, I tried soft rebooting, and then keying the crate again, but the symptoms remained the same. Also, I tried burt restoring to several different times in the last few days, to see if that helped. It didn't. I did notice that MC2 was unhappy, which was a result of the burtrestores setting the MCL filters as if the cavity were locked, so I manually ran mcdown. Also, the MC autolocker script had died, so Kiwamu brought it back to life.
Since we've spent an hour on trying to relock the PSL cavities (the descriptive word I'm going to suggest for us is persistent, not losers), we're giving up in favor of waiting for expert advice in the morning. I suppose there's something obvious that we're missing, but we haven't found it yet......
I checked the situation from my home and the problem was solved.
The main problem was undefined state of the autolocker and the strange undefined switch states, being associated with the bootfest and burtrestore.
- MC UP/DOWN status shows it was up and down. So I ran scripts/MC/mcup and scripts/MC/mcdown. These cleared the MC autolocker status.
- I had a problem handling the FSS. After mcup/mcdown above, I randomly pushed the "enable/disable" buttons and others, and with some reason, it recovered the handling. Actually it acquired the lock autonomously. Kiwamu may have also been working on it at the same time???
- Then, I checked the PSL loop. I disconnected the loop by pushing the "test" button. The DC slider changes the PZT voltage only 0~+24V. This is totally strange and I started pushing the buttons randomly. As soon as I pushed the "BLANK"/"NORMAL" button, the PZT output got back under the control.
- Then I locked the PMC, MZ, and MC as usual.
Alberto: You must be careful as the modulations were restored.
I checked the situation from my home and the problem was solved.
This is a (sort of) known problem with the EPICS computers: it's generally called the 'sticky slider' problem, but of course it applies to buttons as well. It happens after a reboot, when the MEDM control/readback values don't match the actual applied voltages. The solution (so far) is just to `twiddle' the problematic sliders/button. There's a script somewhere called slider_twiddle that does this, but I don't remember if it has PSL stuff in it. A better solution is probably to have an individual slider twiddle script for each target machine, and add the running of that script to the reboot ritual in the wiki.
There was an earthquake, all watchdogs were tripped, ITMX was stuck, and c1psl was dead so MCautolocker was stuck.
Watchdogs were reset (except ETMX which remains shutdown until we finish with the stack weight measurement), ITMX was unstuck using the usual jiggling technique, and the c1psl crate was keyed.
Since there have been various software/hardware activity going on (stack weighing, AUX laser PLL, computing timing errors etc etc), I decided to do a check on the state of the IFO.