The ETMX oplev returning beam is well centered on the qpd. We lost this signal 2 days ago. I will check on the qpd.
ETMX sus damping restored. c1iscex computer is down
The ETMX oplev returning beam is well centered on the qpd. We lost this signal 2 days ago. I will check on the qpd.
ETMX sus damping restored
As reported a couple days ago, the ETMX IO chassis has no timing signal. This is why we there is no QPD signal and why we turned off the ETMX watchdog. In fact, I believe that there is probably nothing coming out of the ETMX suspension controller at all.
I'm working on getting the timing board replaced. Hopefully today.
For some reason or another, I decided that we should see if the optical lever servos were injecting too much noise into the test masses. The ITMs are much worse than the ETMs and I am afeared that they might be making the main noise for our arms in the 20-40 Hz region. Jenne is checking up on these feedback loops to see what's up.
To estimate the actuator gains of the mirrors, I turned on 1 count drives from LSC/CAL oscillators into the LSC drives of each test mass at the frequencies shown in the plot with the resulting peaks showing up in in POX/Y with the single arm locks in red. I will leave these going permanently, but with 0.1 count ampltiudes; we need to make it so in the scripts.
I'm in the process of filling this table
I should replace ETMX He/Ne laser
I took loop measurements of ETMX pit and yaw, and set the upper UGF to be ~6Hz for both. This required a pitch gain of 25, and a yaw gain of 16.
The spectra look similar to what they were before Steve did the swap.
I've turned the ETMX oplev servos off for the time being. (At the input side, so that no scripts will accidently turn it on).
Thus, only the local damping is being applied, let's see if we see any kicks...
During this time period, it looks like there was maybe one excursion. Here are the individual OSEM signals, which I think to be calibrated to microns.
When I'm doing other things, I'm going to intentionally leave the watchdog tripped, and see if anything happens.
During a ETMX kick that just occured, with only local damping on, the slow VMon channels didn't show any noticable change.
How many volts does it take to pitch the ETM by 5 urad?
By looking at the oplev and Vmons before and after a step of 90 counts in SUS-ETMX_PIT_OFFSET, I observe:
UL: +1.53mV / urad
UR: -1.94mV / urad
LR: -1.54mV / urad
LL: +1.92mv / urad
The random error associated with these measurements is ~0.02mV
So, the ~7urad urad shift seen in my earlier post would mean a change of around 10mV in the Vmon signals, which isn't evident in the traces. So, this is possibly a piece of evidence in favor of a real mechanical shift, rather than an electronic glitch.
I believe that the Xend aux laser was turned off earlier today, for Steve's work swapping out the oplev. When I went down there, the red "off" LED was illuminated, and the LCD screen was showing something. I pushed the green "on" button, and I immediately got green.
Also, I saw that the 24Hz roll mode was very rung up on ETMX. I looked at the FM5 "bounce roll" filter, and it had some old values, 12Hz and 18Hz for the resonant gains. All other optics have the proper 16Hz and 24Hz frequencies. I copied the BS oplev bounce roll filter over to ETMX pit and yaw (both were wrong), and loaded them in. The mode is starting to ring down.
UR osem IR shield glass is pushed back. It came out of its clip holder. The magnet is free.
Atm2, UL & LL magnets centered poorly. Almost hinging on opposite sides.
UR & LR centered well. There have plenty of room to move in an earth quake.
I took data of the ETMX SUSPOS, SUSPIT and SUSYAW channels while driving each of the 4 face coils. I manually turned off all the damping except the side.
Excitation: I used white noise bandpassed from 0.4 to 5 Hz in order to examine the responses around the resonance frequencies. To avoid ringing things up too much, I started with a very weak drive signal and gradually increased it until it seemed to have an effect on the mirror motion by looking at the oplev signals/sensor RMS values on the SUS screen; it's possible I'll need to do it again with a stronger signal if there's not enough coherence in the data.
Finding the matrix: The plan is to estimate the transfer function of the coil drive signal with the sensed degrees of freedom (specified by the already diagonalized input matrix). This transfer function can be averaged around the resonance peak for each dof to find the elements of the matrix that converts signals to dof responses, (the "response matrix", which is the inverse of the output matrix). Each column of the response matrix gets normalized so that the degrees of freedom influence the drive signals in the right ratio.
Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.
As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.
The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.
We put ETMX back in its tower, and confirmed its balance. It might be pointing a teensy bit upward, but it is way less than the DC pointing offset we see when we put the OSEMs in the towers (since the PDs and LEDs have some magnetic bits to them).
Discussions are ongoing as to where the ETM should sit on its table, but we'll probably toss it into the chamber later this evening.
I took ETMY out of the magnet gluing fixture, and put it in a ring, in the foil house. It is ready to have the wire winched and get balanced at our convenience.
The updated status table:
ETMX is currently in the clean room, the barrel is the tiniest bit submerged in acetone that will remove a guide rod, standoff, and side OSEM.
Additional inpsection of the standoffs on the flow bench did not provide any insight, pictures are in picasa. Here is a cropped version of a picture we took:
We should look at them under a microscope.
The magnet, guide rod, and standoff came off without too much force. However, some epoxy residue remains on the barrel. I didn't really want to scrape it off, so I've opted for more soaking. Much of the acetone had evaporated already, so I put some more - just to the point where the residue is submerged.
In the evening, I went into the clean room to check how it goes.
- The air around the table is quite warm like a hell. Is this normal?
- I checked how the scattered epoxy spots look like. They were not touching the bath anymore due to evaporation.
- I scraped the spots with the tweezers there. They were easily removed. The particlates on the side barrel were wiped by a wipe with aceton. (Result: Attachment 3)
- Then looked at the other side. I poked the standoff with the tweezer. It was easily removed. I don't think the bond was too weak. Just the area of the bond was so tiny.
- Also residue was scraped by a tweezer and wiped with a cloth. (Attachment 2)
- The removed stand off is in the stainless bowl together with the parts that Eric removed.
- I didn't want to leave the optic in the aceton fume. It was placed on a metal donuts for a 3" optic. (Attachment 4)
- I couldn't find a vacant clean glass jar for the lid. So, a foil hut was built. We should be very careful not to scratch the optic when we remove the hut. (Attachment 5)
- The aceton bath was covered with the foil as it was. (Attachment 6)
Jamie did computer magic. I burt restored scxepics, and restored ETMX damping.
I'm not sure why or when it was tripped, but I have restored the ETMX watchdog.
We have positioned and applied epoxy to one ruby standoff on ETMX, for overnight curing according to the SOS standoff gluing procedure. This included:
Instead of trying to fix up a way of gluing the guiderod with the proper alignment, we chose to be more conservative and glue the standoff today, then switch the gluing fixture's arms tomorrow to glue the guide rod with the good fixture arm.
Additionally, we chose to glue one of the more assymetric standoffs on this first side. What I mean by this is: We have 3 ruby standoffs with grooves. Two of them have the groove about 1/8th of the way along their length, and one has it about 1/4 of the way. Since the second standoff is going to be glued while suspended, after pitch balancing, we figure that we want to use the more centered groove on that side, meaning we used one of the 1/8th standoffs today.
Unfortunately, we neglected to take any pictures :/
Today I took the picture of the glued ruby stand-off. The groove has not been invaded by the epoxy!
Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:
So the ETMX satellite box is unplugged now, starting 530 pm PDT.
I revived the ETMX simplant model, c1spx. It's running on cpu4 on c1iscex, and interfaces with C1scx via SHMEM.
The channel names for the simplant suspensions will be SUP, so the channel from this model will C1:SUP-ETMX_.
Next I'll try to get the ITMX and LSC ("LSP") simplant models running so we can run a "full" cavity simulation.
Sasha has been working on LSP, so we should be ready to do something with that soon. In the mean time she's going to fix up the SPX MEDM screens, since some channel names have been changed since it was last run.
I think the ETMX slow machine might be dead. All of the regular FE readbacks are fine, and the c1iscex FE computer looks fine, but the slow readbacks are all whited out.
I turned off the damping loops for ETMX, since I don't have access to the watchdog disable/enable switch. I guess checking this out will be task #1 for Monday morning.
For lack of a better idea, I keyed the crate. The computer came back up just fine, ETMX is happily damped again.
[Koji, Paco, Yuta]
We implemented the ETMX software watchdog "level 1", meaning the logic disables the SUSPOS/SUSPIT/SUSYAW/OLPIT/OLYAW damping loops whenever the UL/LL/UR/LR/SD PD VAR calculated as soft epics channels in /cvs/cds/caltech/target/c1auxex/ETMXaux.db exceed the PD MAX VAR value stored in C1:SUS-ETMX_PD_MAX_VAR. To do this, we add the following set of record entries to the db file, for example for a single (UL) PD we add:
# Soft Watchdog example from ETMXaux.db
field(DESC,"ETMX UL Sensor Mean")
field(DESC,"ETMX UL Se")
field(DTYP,"Raw Soft Channel")
field("CALC", "0.5*ABS(A - B)")
field(DESC,"ETMX UL Sensor Deviation")
and the logic is implemented for all loops by adding entries like:
# Software Watchdog level #1: turns off SUS damping based on OSEM variance data
field(DESC,"Turn off SUSPOS damping")
field(OUT,"C1:SUS-ETMX_SUSPOS_SW2S PP NMS")
# (...) # All other loops
field(DESC,"Fanout to flip SUS damping switches")
field(LNK1,"C1:SUS-ETMX_POS_DAMP PP NMS")
field(LNK2,"C1:SUS-ETMX_PIT_DAMP PP NMS")
field(LNK3,"C1:SUS-ETMX_YAW_DAMP PP NMS")
field(LNK4,"C1:SUS-ETMX_SID_DAMP PP NMS")
field(LNK5,"C1:SUS-ETMX_OLP_DAMP PP NMS")
field(LNK6,"C1:SUS-ETMX_OLY_DAMP PP NMS")
field(DESC,"Turn off SUS damping")
field(SDIS,"C1:SUS-ETMX_LOGIC PP NMS")
field(OUT,"C1:SUS-ETMX_TRIP_ALL.VAL PP NMS")
Next, we ssh into c1auxex and run systemctl restart modbusIOC.service
If you haven't done so or the signals are not available in dataviewer, you can add them explicitly into /cvs/cds/rtcds/caltech/c1/chans/daq/C0EDCU.ini
Finally, you may have to run sudo systemctl restart rts-edc_c1sus from c1sus
[lydia, steve, ericq, gautam]
[lydia, ericq, gautam]
Lydia also briefly played around with the IR camera to inspect the OSEMs. A more thorough investigation will be done once the cage is in for air baking. From our initial survey, we feel that the beams are pretty well aligned along the straight line between PD and LED - we estimate the upper bound on any misalignment to be ~10 degrees.
Tonight's progress on ETMX:
Since the air bake oven we had been using is out of commision, we're not sure where to do our EP30 test runs. If we are fortunate, we can get the fine pitch balance done tomorrow while Bob is still around, so he can help us quickly bake the test dots, so we can do the standoff gluing.
Here are the photos we took showing the magnet positions in the OSEMs, and others showing the positions of the wire and unglued standoff. These were taken before the pitch balancing adjustment Gautam described, which apparently cause UR to be a little too high. Thoe OSEMs were all inserted only until the ends of the magnets were almost inside, to lower the risk of knocking any magnets off.
At the time of these pictures, all magnets except LL were intentionally positioned slightly above the center of the OSEM in anticipation of wire sag. The LL magnet was approximately centered in the OSEM. It was not possible to get both LL and UL the same height relative to their respective OSEMs, possibly due to a spacing error when they were glued to the optic.
Attachment 1: Position of wire along bottom of the optic. Looks adequately centered and not kinked.
Attachment 2: Photo showing good contact between the sandoff and the barrel of the optic. The standoff does not appear to be resting on glue from the guiderod.
Attachment 3: Shows position of standoff and wire after rough pitch banacing. Wire is visibly resting in the groove.
Attachment 4: SD magnet location photographed through OSEM.
Attachment 5: LL magnet location photographed through OSEM.
Attachment 6: LR magnet location photographed through OSEM.
Attachment 7: UL magnet location photographed through OSEM.
Attachment 8: UR magnet location photographed through OSEM.
Today, we did the following:
I will have another look at the spectra tomorrow morning, to see if the damping improves overnight.
Summary: We did some preliminary tests to check if at least one of the side magnet positions is usable for the side OSEM. We mainly wanted to check how much dynamic range we lose because of the sub-optimal longitudinal positioning of the side magnet. We found that when the side magnet was mainly moving along the axis of the side OSEM (with minimal yaw motion as gauged by eye), the PD voltage bottomed out at ~80 counts (while the completely unoccluded readout was ~800 counts).
Today, we attempted to progress as far as we could towards getting the mirror suspended and gluing the second wire standoff. We think we have a workable setup now. At this stage, the suspension wire has been looped around the magnet, the second wire standoff has been inserted, coarse pitch balancing has been done, and we have verified that side OSEM/magnet positioning is tenable. Details below.
Attachment #3 - Unglued stand off with wire in the groove, mirror freely suspended.
Attachment #4 - Glued stand off with wire in the groove, mirror freely suspended. Clearance between wire and magnet looks reasonable.
Attachment #5 - Barrel of optic (underside), mirror freely suspended. The wire seems to be in a reasonable orientation along the barrel, albeit not perfectly parallel.
Koji just pointed out that we should check that the unglued ruby standoff is in good contact with the barrel of the optic. Attachment #1 suggests that maybe this is not the case. If you zoom into Attachment #1, it is not clear if the standoff is sitting on the glue.
(Full resolution versions of the photos in this ELOG are on picasa)
The OSEM gender changers were not in the box labelled as such, we need these to be able to use the OSEMS to see just how bad the side magnet alignment is, and to do any kind of damping for the fine pitch balancing. The hunt is on.
In the meantime, Gautam and I checked out the standoff seating, and alignment of the face OSEMS (after slightly adjusting the wire length - I guess some sagging is still happening).
With a bit of poking, we convinced ourselves that we sat the standoff in contact with the optic's barrel. Amazingly, we were able to maintain the coarse pitch balance of the optic.
We then partially inserted the face OSEMS, to check their magnet alignment. ("partially" means that the OSEM is not actually enclosing the magnet, we don't want to knock anything off) They seem ok, but not perfect. These magnets were not removed or reglued, so presumably their alignments should be unchanged.
Steve, please look into getting some plated magnets (either SmCo or NdFeB is OK) of this size so that we can install cleaner magnets by the next vent.
ITMY side : Magnet od 1.9 mm so wire to magnet gap ~ 0.2-0.3 mm
After Koji's leap second fix, we were playing around with the X arm locking. In particular, we were playing around with the limit value on the X arm LSC filter bank - the nominal value is 4000, we wanted to see if we could increase this without kicking the optic while acquiring arm lock. We initially increased it to 8000, and then turned it off altogether. Then we rapidly turned the output of the servo ON/OFF, and looked at the arm transmission to see if it came back to the level before unlocking, as an indication of whether the optic was kicked.
These trials suggested a value of 8000 for the limiter was OK, so we left the LSC mode on with the limiter set to 8000. But just as we were about to leave for the night, I noticed on the wall Striptool that the X arm was unlocked. Investigating, we found that the green wasn't even locking to a HOM. Further investigation of the Oplev spot showed that ETMX had received a large kick (both pitch and law errors were ~200urad). ITMX was unaffected.
We initially tried lowering the LSC limit value back to 4000, then used first the Oplev spot and then the green to align the arm. But turning on LSC misaligned the arm after acquiring lock. So we decided to leave LSC off, thinking that the notorious ETMX suspension problems have resurfaced. As a diagnostic, we figured we'd leave the watchdog tripped, and use the Oplev to see if the optic was getting kicked. But the act of turning the watchdog off kicked the optic again (WHY?!).
Looking at the ETMX sus screen, turning off all the damping and LSC (but watchdog on) still leaves a non-zero offset in the "Vmon" field, between 0.02-0.05V depending on the coil. Turning the watchdog OFF takes all these to 0.009V, although I can see the LR value fluctuating between 0.004V and 0.009V. I went to the Xend and squished all the cables on the Sat. Box, but the problem persisted.
At this time, I can't think of any explanation, so I am giving up for the night. To avoid unnecessarily kicking the optic, I am going to unplug the suspension from the Sat. Box and leave one of our tester boxes plugged in, lets see if that sheds any light on the situation...
Last night, I plugged the ETMX suspension coils back into the satellite box. Tonight, we turned on the damping loops for ETMX. Rana centered the Oplev so we can use that as an additional diagnostic to see if the optic gets kicked around overnight. We will re-assess the situation tomorrow.
Sometime earlier today, Lydia noticed that the +/- 5V Sorensens at the X end were not displaying their nominal voltage/current values (as per the stickers on them). She corrected this.
Summary pages show no kicking in the ETMX watchdogs from midnight to 6 AM (0800 - 1400 UTC):
I've gone through the SOS suspension document (E970037) and some old elogs to get an idea of all the accesories we need for the process of suspending, aside from the tower itself, which Steve has already put together. Gautam and I have laid our eyes upon most of the critical pieces. Some other objects are unknown, and perhaps not strictly neccesary.
Confirmed to exist:
In addition, I am told that we have a long ribbon cable that can run from the X end to the clean room to enable OSEM damping control while we do the pitch alignment.
Things mentioned in the procedure I have not found:
Some other tasks and their status:
I've measured ETMX table motion compared to ground motion using accelerometers. Data and settings in the xml file are at the svn directory 40m_seismic/etmx.
<!DOCTYPE LIGO_LW [
<!ELEMENT LIGO_LW ((LIGO_LW|Comment|Param|Time|Table|Array|Stream)*)>
<!ATTLIST LIGO_LW Name CDATA #IMPLIED Type CDATA #IMPLIED>
<!ELEMENT Comment (#PCDATA)>
<!ELEMENT Param (#PCDATA)>
<!ATTLIST Param Name CDATA #IMPLIED Type CDATA #IMPLIED Dim CDATA #IMPLIED
Unit CDATA #IMPLIED>
<!ELEMENT Table (Comment?,Column*,Stream?)>
<!ATTLIST Table Name CDATA #IMPLIED Type CDATA #IMPLIED>
I've measured ETMX table motion compared to ground motion using accelerometers. Data and settings in the xml file are at the svn directory 40m_seismic/etmx
High frequency (>60 Hz) resonances that are present at the ETMX motion spectrum seem to be understandable. Amplification ETMX/GROUND of a factor of 2 at 1 Hz is interesting. I've monitored ACC DQ channels for a few hours and noticed that usually spectrum looks like in the previous elog. But every ~40 min ETMX motion is much higher then ground motion at low frequencies (<5 Hz). I wonder if this a reaction of a table to outside disturbances or accelerometer issue.
Steve has finished installing the enclosure on the new endtable. So Eric and I decided to try and lock the X arm and measure the beam height of the transmitted IR beam relative to the endtable. We initially thought of using POX DC as a the LSC trigger but this did not work as there was no significant change in it when the arm was flashing. Eric then tried misaligning the ITM and using AS110 as a trigger - this worked. We then recompiled the ASS model to take AS110 as an input, and ran the dither alignment. After doing so, I measured the beam height at two points on the new endtable.
So the beam is about 0.7" higher relative to the endtable than we'd like it to be. What do we do about this?
I've also placed two irides extending the cavity axis on the endtable. These should be helpful in aligning the green to the arm eventually.
The new TMC 4' x 3' x4" optical table and enclosure is installed - aligned- leveled.
Atm2, Picture is taken ~42" from the window at 3.75 camera height. The leveled table height is wthin 1/4 at the center of the window.
I think this is close enough to move on with the installation of the optics.
We can raise the loaded table in the future if it is needed.
Atm4, Optical table height to floor 33" at the south west corner
Atm3, Enclosure top cover transmission at 1064 nm, 1mm beam size, power level 157 mW, 0 degree incident angle, T 1.3% Metal shield is required above 100 mW hitting the wall of the enclosure!
Atm5, window to enclosure Kapton seal
Optical layout of the current endtable at ETMX has been updated in the svn repository (directory: 40M_Optical Layout). This layout will help in redesigning the table for the proposed replacement.
Some part numbers of mounts/optics are missing and will be updated once I find them. If you find anything wrong with the layout, do let me know.
We believe the optimal OSEM damping would use an input matrix diagonalized to the free swing modes of the optic, and an output matrix which drives the coils appropriately to damp these free swing modes. As was discovered, a free swinging optic does not necessarily have eigenmodes that match up perfectly with pitch and yaw, however in the current state the "TO_COIL" output matrix that determines the drive signals in response to the diagonlized sensor output also controls the drive signals for the oplevs, LSC/ASC, and alignment biases. So attempts to diagonalize the output matrix to agree with the input matrix have resulted in problems elsewhere. (See previous elog). So, we want to expand the "TO_COIL" matrices to treat the OSEM sensor inputs separately from the others.
...will be helpful for acquiring lock after the vent. We should install a camera at ETMX.
The daytime crew had noticed that there were some ETMX angular shifts happening without any control or intention.
I reseated and strain relieved the bias cable coming into the backplane of the coil driver and now it seems OK.
In the 4-hour-long second trend plot below, the era before 2300 is before reseating. Afterwards, we make a couple adjustments, but so far there has been no un-asked for alignment shifts.
AdS has been run on both arms and offsets saved. Its locked on green/red and the beat frequencies are low and the amplitudes high.
Sun May 18 23:53:37 2014: still OK...I declare it fixed.