** The notation here is [UL, UR, LR, LL]
I turned it back on, maybe around 11am? Definitely a little while before the 12:30 meeting.
EDIT by KI:
Sorry, it's me. I was checking if AC was doing something bad on the ALS noise.
Koji and I wanted to turn off the IFO-room AC so the wind would not blow on MC1-3. We could not. The switches were probably bypassed when the power transformer was replaced at the last scheduled power outage.
There is one three position manual/off/auto switch next to the filter for each unit at CES. They have to be in AUTO position when we want to turn AC on/off from the lab.
ETMY sus damping was disabled. Green locking laser and associated electronics turned off. Computers and power supplies turned off at rack 1Y4
The electricians picking up ac power from 1Y4 manual disconnect box and installing conduit line to ISCT-ETMY east end optical table.
There will be no more daisy chaining this way.
The power is back on at ETMY . c1iscey has not been restarted.
Now I'm turning ac power off at ETMX for the same job to be done.
The power was turned back on at 4pm It took some time for Suresh to restart the computers. We have damping but things are not perfect yet. Auto BURTH did not work well.
I was notified by CIT Utilities that there was a power surge or short power outage this after noon.
Lab conditions are normal: c1ioo is down. The south arm AC was off......I turned it back on.
CALIFORNIA INSTITUTE OF TECHNOLOGY
UTILITY & SERVICE INTERRUPTION
Date: Saturday, June 23, 2012
Time: 3:46 P.M.
Interruption: Electrical Power Disturbance
Contact: Tom Brennan, x-625-395-4984
*The City of Pasadena Water & Power Department had a 34,000-volt line event on Saturday June 23 at 3:46 p.m. This caused a city wide disturbance on the power grid. The Campus did not lose electrical power. However, the disturbance may have affected sensitive electronic equipment.
(If there is a problem with this Interruption, please notify the Service Center X-4717 or the above Contact as soon as possible.
If no response is received we will proceed with the interruption.)
Interim Director of Campus Operations & Maintenance
Koji, Manasa, Jenne, Jamie, Bob and Steve
Access connector removed this morning and work has began in the IOO chamber. BE WARE OF ANTS !
Manasa, Eric, Evan, Koji and Steve,
Access connector removed in order to complete alignment. Light aluminum with acetate windows AC installed.
Prior to the access connector removal, Manasa and I aligned the IFO mirrors.
The arms were locked and aligned by ASS.
The air cond was off for 2 hrs. I just switched it back on at 15:51
I have installed ACAD 2002 on one of the Windows machines in the Control Room. It is on the machine which has Solid Works (called C21530).
The installation files are in MyDocuments under Acad2002. This a shared LIGO license which Christian Cepada had with him.
I hope we will be able to open our optical layout diagrams with this and update them even though it is an old version.
ADC 3 INPUT 4 (#3 in the c1pem model if you count from 0) is bad. It adds DC = ~1 V to the signal as well as noise. I plugged in GUR2 channels to STS1 channels (7-9).
5x 16bit ADC adapter boards (D0902006) assembled.
Since the classification finally works (or seems to work..), I wrote triangulation scripts in Python which triangulate the signals, and a plotting script in Matlab which generates a heat map of seismic noise source locations. I switched the ADC Streckeisen and Trillium connections in order to better triangulate with the current channels, and will return them either tomorrow, or when I come back from Livingston so that we can have weekday data as well.
connector J9B of hardware ADC --> ch1 in software ADC --> GCY_ERR
connector J14 of hardware ADC --> ch11 in software ADC --> GCY_PZT
connector J15 of hardware ADC --> ch13 in software ADC --> GCY_REFL_DC
Yesterday I wired the outputs from the seismometers directly to the ADC input bypassing the old AA board circuit as is described in this elog. The old circuit converted the single-ended output from the seismometers to a differential signal. Today I looked at whether 60 Hz noise is worse going directly into the ADC due to the loss of the common mode rejection previously provided by the conversion to differential signals.
I split the output from the BS Z seismometer to the new board and to an SR785. On the SR785 I measured the difference between the inner and outer conductors of the seismometer output, i.e. A-B with A the center conductor and B the outer conductor, with grounded input. At the same time I took a DTT spectrum of C1:PEM-SEIS_BS_Z_IN1. Both spectra were taken with 1 Hz bandwidth and 25 averages. The setup is shown in attachment 1.
The spectra are shown in attachment 2. The DTT spectrum was converted from counts to volts by multiplying by 2 * 10 V/32768 cts where the extra factor of 2 is from converting from single-ended to differential input. If there was common 60 Hz noise that the ADC was picking up we would expect to see less noise at 60 Hz in the SR785 spectrum measured directly at the output from the seismometer since that was a differential measurement. Since both spectra have the same 60 Hz noise, this noise is differential.
As described in this elog, the ADC for the seismometers now has the signals wired directly to the ADC instead of going through an AA board or other circuit to remove any common mode noise. This elog describes one test of the common mode rejection of this setup. Guantanamo suggested comparing directly with a recent spectrum taken a few months before the new setup described in this elog.
Today I took a spectrum (attachment 1) of C1:PEM-MIC_2 (Ch17) and C1:PEM-MIC_3 (Ch18) with input to the ADC terminated with 50 Ohms. These are two of the channels plotted in the previous spectrum, though I don't know how that plot was normalized. It's clear that there are now strong 60 Hz harmonic peaks that were not there before, so this new setup does have worse common mode rejection.
ADC noise is not a limiting noise source in a current ALS setup.
Below is the calibrated spectrum of C1:ALS-COARSE_I_ERR when
Y arm swinging with just damping (red; taken last night)
terminated before AA (green)
blocked PSL green beam (blue)
Blue and green curve tells us that noise from the beat PD to ADC is not contributing to the Y arm length sensing noise.
Gautam and I measured the noise of the ADC for channels 17, 18, and 19. We plan to use those channels for measuring the noise of the temperature sensors, and we need to figure out whether or not we will need whitening and if so, how much. The figure below shows the actual measurements (red, green and blue lines), and a rough fit. I used Gautam's elog here and used the same function, (with units of nV/sqrt(Hz)) to fit our results. I used a = 1, b = 1e6, c = 2000. Since we are interested in measuring at lower frequencies, we must whiten the signal from the temperature sensors enough to have the ADC noise be negligible.
We want to be able to measure to accuracy at 1Hz, which translates to about current from the AD590 (because it gives ). Since we have a 10K resistor and V=IR, the voltage accuracy we want to measure will be . We would need whitening for lower frequencies to see such fluctuations.
To do the measurements, we put a BNC end cap on the channels we wanted to measure, then took measurements from 0-900Hz with a bandwidth of 0.001Hz. This setup is shown in the last two attachments. We used the ADC in 1X7.
I ceated a simple circuit that takes in 15V and outputs precisely 5V by using a 12V voltage regulator LM7812 and an AD586 that takes the output of the voltage regulator and outputs 5V (attachment 1). We plugged this into the slow channel and will leave it running for a few hours to see if we still have the fluctuations we observed earlier and also fit the noise curve. We'll also test the fast channel later as well. Attachment 2 shows the setup we have in the lab, with the red and white cable plugged into the +15V power supply and the red and black cable connected to the slow channel.
I shorted the inputs on three channels and the outputs on three channels of the Guralp box, and I did similar things with the accelerometers. I was going to move the instruments themselves back, but I didn't have time, so they are still in the box in the corner. If the setup could stay as-is for at least a few hours, that would be awesome.
Todd E. came by this morning and gave us (i) 1x new ADC card and (ii) 1x roll of 100m (2017 vintage) PCIe fiber. This afternoon, I replaced the old ADC card in the c1lsc expansion chassis, and have returned the old card to Todd. The PCIe fiber replacement is a more involved project (Steve is acquiring some protective tubing to route it from the FE in 1X6 to the expansion chassis in 1Y3), but hopefully the problem was the ADC card with red indicator light, and replacing it has solved the issue. CDS is back to what is now the nominal state (Attachment #1) and Yarm is locked for Jon to work on his IFOcoupling study. We will monitor the stability in the coming days.
(i) to replace the old generation ADC card in the expansion chassis which has a red indicator light always on and (ii) to replace the PCIe fiber (2010 make) running from the c1lsc front-end machine in 1X6 to the expansion chassis in 1Y3, as the manufacturer has suggested that pre-2012 versions of the fiber are prone to failure. We will do these opportunistically and see if there is any improvement in the situation.
Looks like the ADC was not to blame, same symptoms persist.
The PCIe fiber replacement is a more involved project (Steve is acquiring some protective tubing to route it from the FE in 1X6 to the expansion chassis in 1Y3), but hopefully the problem was the ADC card with red indicator light, and replacing it has solved the issue.
Gautam and I restarted the models on c1lsc, c1ioo, and c1sus. The LSC system is functioning again. We found that only restarting c1lsc as Rolf had recommended did actually kill the models running on the other two machines. We simply reverted the rebootC1LSC.sh script to its previous form, since that does work. I'll keep using that as required until the ongoing investigations find the source of the problem.
we did a bunch of tests to figure out the feasibility of the plan I outlined last night. Bottom line is: we appear to have a working 64 channel ADC (but with differential receiving that means 32 channels). But we need an aLIGO ADC adaptor card (I'm not sure of the DCC number but I think it is D0902006). See attached screenshot where we managed to add an ADC block to the IOP model on c1lsc, and it recognizes the additional ADC. The firmware on the (newly installed) working card is much newer than that on the existing card inside the expansion chassis (see Attachment #1).
Note that we have left the working ADC card inside the c1lsc expansion chassis. Plan is to give Rolf the faulty ADC card and at the same time ask him for a working adapter board.
Unrelated to this work: we have also scavenged 4 pcs of v2 of the differential receiving AA board from WB EE shop, along with a 1U chassis for the same. These are under my desk at the 40m for the moment. We will need to re-stuff these with appropriate OpAmps (and also maybe change some Rs and Cs) to make this board the same as v6, which is the version currently in use.
We checked back in time to see how the BS and PRM OSEM slow channels are zero. It was clear that they became zero when we worked on this issue on June 17th, Thursday. So we simply went back and power cycled the c1susaux acromag chassis. After that, we had to log in to c1susaux computer and run
sudo /sbin/ifdown eth1
sudo /sbin/ifup eth1
This restarted the ethernet port acromag chassis is connected to. This solved this issue and we were able to see all the slow channels in BS and PRM.
But then, we noticed that the OPLEV of ITMX is unable to read the position of the beam on the QPD at all. No light was reaching the QPD. We went in, opened the ITMX table cover and confirmed that the return OPLEV beam is way off and is not even hitting one of the steering mirrors that brings it to the QPD. We switched off the OPLEV contribution to the damping.
We did burt restore to 16th June morning using
burtwb -f /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2021/Jun/16/06:19/c1susaux.snap -l /tmp/controls_1210622_095432_0.write.log -o /tmp/controls_1210622_095432_0.nowrite.snap -v
This did not solve the issue.
Then we noticed that the OSEM signals from ITMX were saturated in opposite directions for Left and Right OSEMs. The Left OSEM fast channels are saturated to 1.918 um for UL and 1.399 um for LL, while both right OSEM channels are bottomed to 0 um. On the other hand, the acromag slow PD monitors are showing 0 on the right channels but 1097 cts on UL PDMon and 802 cts in LL PD Mon. We actually went in and checked the DC voltages from the PD input monitor LEMO ports on the ITMX dewhitening board D000210-A1 and measured non-zero voltages across all the channels. Following is a summary:
We even took out the 4-pin LEMO outputs from the dewhitening boards that go to the anti-aliasing chassis and checked the voltages. They are same as the input voltages as expected. So the dewhitening board is doing its job fine and the OSEMs are doing their jobs fine.
It is weird that both the ADC and the acromags are reading these values wrong. We believe this is causing a big yaw offset in the ITMX control signal causing the ITMX to turn enough make OPLEV go out of range. We checked the CDS FE status (attachment 1). Other than c1rfm showing a yellow bar (bit 2 = GE FANUC RFM card 0) in RT Net Status, nothing else seems wrong in c1sus computer. c1sus FE model is running fine. c1x02 (the lower level model) does show a red bar in TIM which suggests some timing issue. This is present in c1x04 too.
Currently, the ITMX coil outputs are disabled as we can't trust the OSEM channels. We're investigating more why any of this is happening. Any input is welcome.
After sliding the alignment bias around and browsing through elog while searching for "stuck" we concluded the ITMX osems needed to be freed. To do this, the procedure is to slide the alignment bias back and forth ("shaking") and then as the OSEMs start to vary, enable the damping. We did just this, and then restored the alignment bias sliders slowly into their original positions. Attachment 1 shows the ITMX OSEM sensor input monitors throughout this procedure.
At the end, since MC has trouble catching lock after opening PSL shutter, I tried running burt restore the ioo to 2021/Jun/17/06:19/c1iooepics.snap but the problem persists
I carried some further modifications and tests to the AI Board. Details and observations here:
I think the board is okay to be used now.
I tried shifting the notch frequencies on the D000186-revision D board given to me by Koji. The existing notches were at ~16 kHz and ~32 kHz. I shifted these to notches at ~64 kHz and ~128 kHz by effecting the following changes (see schematic for component numbering) on Channel 8 of the board-I decided to check things out on one channel before implementing changes en masse:
=> New notches should be at 66.3 kHz and 131.7 kHz.
I then measured the frequency response of the modified channel using the SR785, and compared it to the response I had measured before switching out the resistors. The SR785 only goes up to 102 kHz, so I cannot verify the 128 kHz notch at this point. The position of the 64 kHz notch looks alright though. I think I will go ahead and switch out the remaining resistors in the evening.
Note 1: These plots are just raw data from the SR785, I have not tried to do any sort of fitting to poles and zeros. I will do this at some point.
Note 2: All these smts were taken from Downs. Todd helped me locate the non-standard value resistors. I also got a plastic 25-pin D-sub backshells (the spares are in the rack), with which I have fashioned the required custom ribbon cables (40 pin IDC to 25 pin D-sub with twisted ribbon wire, and a short, 10pin IDC to 10pin IDC with straight ribbon wire).
When signals are transmitted between the models running at different rates, no AI or AA filters are automatically applied. We need to fix our models.
[ Yuki, Gautam ]
I improved Anti-Imaging board (D000186-Rev.D), which will be put between DAC port and PZT driver board.
It had notches at f = 16.6 kHz and 32.7 kHz, you can see them in the plot attached. So I replaced some resistors as follows:
Then the notch moved to 65.9 kHz (> sampling frequency of DAC = 64 kHz, good!).
(The plot enlarged around the notch frequency and the plot of all channels will be posted later.)
All electronics and optics seem to be ready.
I made a cable which connects DAC port (40 pins) and AI board (25 pins). I will check if it works.
Tomorrow I will change setup for improvement of AUX Y-end green locking. Any optics for IR will not be moved in my design, so this work doesn't affect Y-arm locking with main beam.
While doing this work, I will do:
I updated the mime.local.conf file for the AIC Wiki so as to allow attachments with the .txz format. THis should be persistent over upgrades, since its a local file.
We locked the XARM and YARM with using ALS control loop and we succeeded to lock stably both arms. The performance of the ALS was tested with a measurement of the calibrated error signal. (attachment 1)
- red and blue : the in-loop noise of ALS of each arm.
- green and purple:Stability of the beat-note frequency with the MC and the arm freely running.
In the high frequency region, YARM has larger noise than XARM, and these noises were not there in previous measurements by Koji and Manasa (elog8865). You can see that in both of in-loop noise and free running noise. These noises may be caused by the Green PDH servo or hte phase tracker servo or any other electrical staff. We will start noise budget of these servo.
At higher frequency than UGF of ASL control loop, the loop does not suppress the noises at all, but the inloop and free running noise are not equivalent. I have no idea about that so far.
What was the beat freq for each arm?
The HF noise level depends on the frequency of the beat note.
As the BBPD has the freq dependent noise level. (See this entry)
What was the beat freq for each arm?
The HF noise level depends on the frequency of the beat note.
As the BBPD has the freq dependent noise level. (See this entry)
I'm not sure about the actual number of the beat frequency, but the beat frequency was almost same in both arms. And I took this measurement sometimes with slightly different beat frequency but the noise level didn't change so much.
It was known that the Y end ALS PZTs are not working. But Anchal reported in the meeting that the X end PZTs are not working too.
We went down to the X arm in the afternoon and checked the status. The HV (KEPCO) was off from the mechanical switch. I don't know this KEPCO has the function to shutdown the switch at the power glitch or not.
But anyway the power switch was engaged. We also saw a large amount of misalignment of the X end green. The alignment was manually adjusted. Anchal was able to reach ~0.4 Green TRX, but no more. He claimed that it was ~0.8.
We tried to tweak the SHG temp from 36.4. We found that the TRX had the (local) maximum of ~0.48 at 37.1 degC. This is the new setpoint right now.
ALS common locked by actuating on MC2 and ALS Differential locked by actuating on ETMX and ETMY (Stable lock acquired for over an hour).
Common and Differential offsets were swept to obtain IR resonance in both the arms (arms stayed on resonance for over 15 minutes).
1. Configured LSC settings to allow locking using ALS error signals.
2. Locked common and differential using ALS error signals
XARM 1 -1
YARM 1 1
X arm servo settings:
FIlters: FM1, FM5, FM6, FM7, FM9
Gain = -8.0
Y arm servo settings:
Filters: FM1, FM5, FM6, FM7, FM9
Gain = +8.0
ETMX 1 0
ETMY 0 1
3. Transitioned CARM control output to actuate on MC2 instead of ETMX
SUS-MC2_LSC servo gain = 1.0
The transition was done in very small steps : actuating on MC2 in -0.01 steps at the outmatrix upto -1.0 while reducing the ETMX actuation to 0 simultaneously.
DARM still stayed locked only with actuation on ETMY.
4. Transitioned DARM control to ETMX and ETMY.
Used ezcastep to step up DARM control (Y arm output) actuation on ETMX and step down the actuation on ETMY.
Final output matrix
ETMX 0 -0.5
ETMY 0 0.5
MC2 -1.0 0
Noise plot in attachment.
5. Finding arm resonance
Used ezcastep to gradually build up offsets in CARM (LSC-XARM_OFS) to find IR resoance in one arm (Y arm).
Introducing a small (order of 0.5) DARM offset (LSC-YARM_OFS) shifted the Y arm off-resonance.
Used CARM offset to get back the Y arm to resonance.
Changing CARM and DARM offsets alternately while tracking the Y arm resonance got us to a point where we had both the arms resonating for IR.
6. At this point the MC decided to give up and we lost lock.
1. We found that the WFS2 YAW output filterbank had the output switched OFF (probably accidentally by one of us). This was reenabled. Please be careful while manually turning ON and OFF the MC WFS servos.
The IFO is being uncooperative tonight, and I have an early morning meeting, so I'm calling it a night.
Koji's filter module changes have been propagated from the Xarm to the Yarm, to CARM and to DARM. (Actually, Q overwrote the changes to Xarm on Sunday accidentally, so first he reverted those for us, and then we propagated the changes).
Today, with careful measuring, we find that for X and Y arms individually locked with the ALS, we want the gains to be +17 for the Yarm, and -17 for the Xarm (with the beatnote up-is-up convention). This puts the UGFs at 150 Hz.
We then switched over to CARM and DARM locking. We guessed that the gains should be a factor of 2 lower since we're pushing on both ETMs for DARM, and the MC2 actuator is roughly the same strength as the sum of the ETMs. In the end, after measuring the CARM and DARM loops, we find that the gains should be +7.5 for CARM, and +8.0 for DARM to set the UGFs at 150 Hz. The servo is a little bit delicate, so having too low of gain is not okay.
For some reason, we seem to be utilizing more actuator range with the new setup, so the limiters in the filter banks have been set to 11,000 (previously were 8,000), and the ALS watch script (ALSdown.py) threshold has been increased to 10,000 (previously 7,000).
When finding the IR resonances with the new scheme, we are having trouble holding lock throughout the scan. I have set the tramp for the coarse part of the scan to be 0.05 seconds (previously 0.01 seconds), which is an increase of a factor of 5 in the ramp time. This helps, but may still not be enough, since we don't always hold lock until both IR resonances are found.
Probably the most annoying thing from tonight is the fact that ETMY keeps drifting off, particularly in yaw, when locked. I don't have an explanation of why this is happening, but you can watch it happen sometimes, and the lock will be lost shortly thereafter. Definitely when we lose lock and the ETM gets kicked, it is far enough away in yaw alignment that I have to completely redo the Yarm alignment. This happens whether or not the ETMY oplevs are on.
To summarize, 3 scripts have been modified:
(1) ALSdown - threshold increased (Modification from last week - turns off the slow temp servos for the end lasers, clears histories)
(2) ALSfindIRresonance - increase ramp time
(3) Lock_ALS_CARMandDARM - final gain values set to 7.5 for CARM and 8 for DARM, no filters come on until gains all the way up, turns on new set of Koji filters. (Modification from last week - turns on the slow temperature servos for the end lasers)
We've been having trouble tuning the ALS DIFF matrix. Trying to see if the MC2 EXC can be cancelled in ALS DARM by adjusting the relative gains in ALSX and ALSY Phase Tracker outputs.
There's a bunch of intermittent behavior. Between different ALS locks, we get more or less cancellation. We were checking this by driving MC2 at ~100-400 Hz and checking the ALS response (with the ALS loops closed). We noticed that the X and Y readbacks were different by ~5-10 degrees and that we could not cancel this MC2 signal in DARM by more than a factor of 4-5 or so. In the middle of this, we had one lock loss and it came back up with 100x cancellation?
Attached is a PDF showing a swept sine measurement of the ALSX, ALSY, and DARM signals. You can see that there is some phase shift between the two repsonses leading to imperfect cancellation. Any ideas? Whitening filters? HOM resonance? Alignment?
The front panels for the ALS delay line box came in last week. Some of the holes for the screws were slightly misaligned, so I filed those and everything is now put together. I just need to test both front panels to determine if the SMAs should be isolated or not.
Koji had also suggested making the holes in the front and back panel conical recesses so that flat head screws could be used and would counteract the anodization of the panel and avoid the SMAs being isolated. I think if we did that then conductivity would be ensured throughout the panel and also through the rest of the box. I also think one way we could test this before drilling conical recesses would be to test both front panels now, as one has isolated SMAs and one has conductive SMAs. If the anodization of the panel isolated the SMA regardless, we could potentially figure this out by testing both panels. But, would it also be that it is possible that the isolation of the SMA itself does not matter and so this test would tell us nothing? Is there a better way to test if the SMAs are being isolated or not? Or would this be more time consuming than just drilling conical recesses as a preventative measure?
I tested both of the front panels (conductive and isolated SMAs) with the ALS Delay Line Box by driving extremely close frequencies through the cables. By doing this, we would expect that a spike would show up in the PSD if there was crosstalk between the cables.
In the plots below, for the conductive panel, the frequencies used were
X Arm: 22.329 MHz Y Arm: 22.3291 MHz
For the isolated panel, the frequencies were
X Arm: 22.294 MHz Y Arm: 22.2943 MHz
This gives a difference of 100 Hz for the conductive panel and 300 Hz for the isolated panel. Focusing on these areas of the PSD, it can be seen that in the Y Arm cable there is a very clear spike within 30 Hz of these differences when frequencies are being driven through both cables as opposed to the signal being in only the Y Arm. In the X Arms, the noise in general is higher when both cables are on, but there is no distinct spike at the expected frequencies. This indicates that some sort of crosstalk is probably happening due to the strong spikes in the Y Arm cables.