[steve, koji, gautam]
We took another pass at this today, and it seems to have worked - see Attachment #1. I'm leaving CDS in this configuration so that we can investigate stability. IMC could be locked. However, due to the vacuum slow machine having failed, we are going to leave the PSL shutter closed over the weekend.
Steve pointed out that some of the vacuum MEDM screen fields were reporting "NO COMM". Koji confirmed that this is a c1vac1 problem, likely the same as reported here and can be fixed using the same procedure.
However, Steve is worried that the interlock won't kick in in case of a vacuum emergency, so we are leaving the PSL shutter closed over the weekend. The problem will be revisited on Monday.
Multiple realtime processes on c1sus are suffering from frequent time outs. It eventually knocks out c1sus (process).
Obviously this has started since the fiber swap this afternoon.
gautam 10pm: there are no clues as to the origin of this problem on the c1sus frontend dmesg logs. The only clue (see Attachment #3) is that the "ADC" error bit in the CDS status word is red - but opening up the individual ADC error log MEDM screens show no errors or overflows. Not sure what to make of this. The IOP model on this machine (c1x02) reports an error in the "Timing" bit of the CDS status word, but from the previous exchange with Rolf / J Hanks, this is down to a misuse of ADC0 Ch31 which is supposed to be reserved for a DuoTone diagnostic signal, but which we use for some other signal (one of the MC suspension shadow sensors iirc). The response is also not consistent with this CDS manual - which suggests that an "ADC" error should just kill the models. There are no obvious red indicator lights in the c1sus expansion chassis either.
We had another crash of c1sus and Gautam did full power cycling of c1sus. It was a sturggle to recover all the frontends, but this solved the timing issue.
We went through full reset of c1sus, and rebooting all the other RT hosts, as well as daqd and fb1.
[ Yuki, Koji, Gautam ]
An alignment of AUX Y end green beam was bad. With Koji and Gautam's advice, it was recovered on Friday. The maximum value of TRY was about 0.5.
Following the procedure in this elog, we effected a reset of the vacuum slow machines. Usually, I just turn the key on these crates to do a power cycle, but Steve pointed out that for the vacuum machines, we should only push the "reset" button.
While TP1 was spun down, we took the opportunity to replace the TP1 controller with a spare unit the company has sent us for use while our unit is sent to them for maintenance. The procedure was in principle simple (I only list the additional ones, for the various valve closures, see the slow machine reset procedure elog):
However, we were foiled by a Philips screw on the DB37 connector labelled "MAG BRG", which had all its head worn out. We had to make a cut in this screw using a saw blade, and use a "-" screwdriver to get this troublesome screw out. Steve suspects this is a metric gauge screw, and will request the company to send us a new one, we will replace it when re-installing the maintaiend controller.
Attachments #1 and #2 show the Vacuum MEDM screen before and after the reboot respectively - evidently, the fields that were reading "NO COMM" now read numbers. Attachment #3 shows the main volume pressure during this work.
The problem will be revisited on Monday.
Precondition: c1vac1 & c1vac2 all LED warning lights green [ atm3 ], the only error message is in the gauge readings NO COMM, dataviewer will plot zero [ atm1 ], valves are operational
When our vacuum gauges read " NO COMM " than our INTERLOCKS do NOT communicate either.
So V1 gate valve and PSL output shutter can not be triggered to close if the the IFO pressure goes up.
[ only CC1_HORNET_PRESSURE reading is working in this condition because it goes to a different compuer ]
I've been plugging away at Altium prototyping the high-voltage bias idea, this is meant to be a progress update.
I need to get footprints for some of the more uncommon parts (e.g. PA95) from Rich before actually laying this out on a PCB, but in the meantime, I'd like feedback on (but not restricted to) the following:
I also don't have a good idea of what the PCB layer structure (2 layers? 3 layers? or more?) should be for this kind of circuit, I'll try and get some input from Rich.
*Updated with current noise (Attachment #2) at the output for this topology of series resistance of 25 kohm in this path. Modeling was done (in LTspice) with a noiseless 25kohm resistor, and then I included the Johnson noise contribution of the 25k in quadrature. For this choice, we are below 1pA/rtHz from this path in the band we care about. I've also tried to estimate (Attachment #3) the contribution due to (assumed flat in ASD) ripple in the HV power supply (i.e. voltage rails of the PA95) to the output current noise, seems totally negligible for any reasonable power supply spec I've seen, switching or linear.
We have been working on double checking the noise budget calculations. We wanted to evaluate the amount of squeezing for a few different scenarios that vary in cost and time. Here are the findings:
All calculations done with
Main unbudgeted noises:
Threat matrix has been updated.
This is the procedure I follow when I take these measurements for the XARM (symmetric under XARM <-> YARM):
Information for the armloss measurement:
Note: The scripts uses httplib2 module. You have to install it if you don't have.
The locked arms are needed to calculate armloss but the alignment of PMC is deadly bad now. So at first I will make it aligned. (Gautam aligned it and PMC is locked now.)
gautam: The PMC alignment was fine, the problem was that the c1psl slow machine had become unresponsive, which prevented the PMC length servo from functioning correctly. I rebooted the machine and undid the alignment changes Yuki had made on the PSL table.
Gautam and Steve,
Our TP3 drypump seal is at 360 mT [0.25A load on small turbo] after one year. We tried to swap in old spare drypump with new tip seal. It was blowing it's fuse, so we could not do it.
Noisy aux drypump turned on and opened to TP3 foreline [ two drypumps are in the foreline now ] The pressure is 48 mT and 0.17A load on small turbo.
With Gautam's help, Y-arm was locked. Then I ran the script "armloss_dcrefl_asdcpd_scope.py" which gets the signals from oscilloscope. It ran and got data, but I found some problems.
Anyway, I got the data needed so I will calculate the loss after converting the format.
We want to measure the pressure gradient in the 40m IFO
Our old MKS cold cathodes are out of order. The existing working gauge at the pumpspool is InstruTech CCM501
The plan is to purchase 3 new gauges for ETMY, BS and MC2 location.
Basic cold cathode or Bayard-Alpert Pirani
I ran the script for measuring arm-loss and calculated rough Y-arm round trip loss temporally. The result was 89.6ppm. (The error should be considered later.)
The measurement was done as follows:
('AS_DARK =', '0.0019517200000000003') #dark noise at ASDC
('MC_DARK =', '0.02792') #dark noise at MC2 trans
('AS_LOCKED =', '2.04293') #beam power at ASDC when the cavity was locked
('MC_LOCKED =', '2.6951620000000003')
('AS_MISALIGNED =', '2.0445439999999997') #beam power at ASDC when the cavity was misaligned
('MC_MISALIGNED =', '2.665312')
#normalized beam power
the script "armloss_AS_calc.py",
Some changes were made in the script for getting the signals of beam power:
In the yesterday measurement the beam power of ASDC is higher when locked than when misaligned and I wrote it maybe caused by over-coupled cavity. Then I did a calculation as following to explain this:
DASWG is not what we want to use for config; we should use the K. Thorne LLO instructions, like I did for ROSSA.
pianosa has been upgraded to SL7. I've made a controls user account, added it to sudoers, did the network config, and mounted /cvs/cds using /etc/fstab. Other capabilities are being slowly added, but it may be a while before this workstation has all the kinks ironed out. For now, I'm going to follow the instructions on this wiki to try and get the usual LSC stuff working.
I used these values for measuring armloss:
then the uncertainties reported by the individual measurements are on the order of 6 ppm (~6.2 for the XARM, ~6.3 for the YARM). This accounts for fluctuations of the data read from the scope and uncertainties in mode-matching and modulation depths in the EOM. I made histograms for the 20 datapoints taken for each arm: the standard deviation of the spread is over 6ppm. We end up with something like:
XARM: 123 +/- 50 ppm
YARM: 152+/- 50 ppm
This result has about 40% of uncertaintities in XARM and 33% in YARM (so big... ).
In the previous measurement, the fluctuation of each power was 0.1% and the fluctuation of P(Locked)/P(misaligned) was also 0.1%. Then the uncertainty was small. On the other hand in my measurement, the fluctuation of power is about 2% and the fluctuation of P(Locked)/P(misaligned) is 2%. That's why the uncertainty became big.
We want to measure tiny value of loss (~100ppm). So the fluctuation of P(Locked)/P(misaligned) must be smaller than 1.6%.
(Edit on 10/23)
I think the error is dominated by systematic error in scope. The data of beam power had only 3 degits. If P(Locked) and P(misaligned) have 2% error, then
You have to check the configuration of scope.
but there's one weirdness: It get's the channel offset wrong. However this doesn't matter in our measurement because we're subtracting the dark level, which sees the same (wrong) offset.
When you do this measurement with oscilloscope, take care two things:
Steve & Bob,
Bob removed the head cover from the housing to inspect the condition of the the tip seal. The tip seal was fine but the viton cover seal had a bad hump. This misaligned the tip seal and it did not allow it to rotate.
It was repositioned an carefully tithened. It worked. It's starting current transiant measured 28 A and operational mode 3.5 A
This load is normal with an old pump. See the brand new DIP7 drypump as spare was 25 A at start and 3.1 A in operational mode. It is amazing how much punishment a slow blow ceramic 10A fuse can take [ 0215010.HXP ]
In the future one should measure the current pick up [ transient <100ms ] after the the seal change with Fluke 330 Series Current Clamp
It was swapped in and the foreline pressure dropped to 24 mTorr after 4 hours. It is very good. TP3 rotational drive current 0.15 A at 50K rpm 24C
The scripts for measuring armloss are in the directory "/opt/rtcds/caltech/c1/scripts/lossmap_scripts/armloss_scope".
The main laser went off when PSL doors were opened-closed. It was turned back on and the PSL is locked.
As a part of the preparation for the replacement of c1susaux with Acromag, I made inspection of the coil-osem transfer function measurements for the vertex SUSs.
The TFs showed typical f^-2 with the whitening on except for ITMY UL (Attachment 1). Gautam told me that this is a known issue for ~5 years.
We made a thorough inspection/replacement of the components and identified the mechanism of the problem.
It turned out that the inputs to MAX333s are as listed below.
The switching voltage for UL is obviously incorrect. We thought this comes from the broken BIO board, and thus swapped the corresponding board. But the issue remained. There are 4 BIO boards in total on c1sus, so maybe we have replaced a wrong board?
Initially we thought that the BIO can't drive the pull-up resister of 5KOhm form 15V to 0V (=3mA of current). So I have replaced the pull-up resisters to 30KOhm. But this did not help. These 30Ks are left on the board.
Gautam & Steve,
Our controller is back with Osaka maintenace completed. We swapped it in this morning.
TP-1 Osaka maglev controller [ model TCO10M, ser V3F04J07 ] needs maintenance. Alarm led on indicating that we need Lv2 service.
The turbo and the controller are in good working order.
Our maintenance level 2 service price is $...... It consists of a complete disassembly of the controller for internal cleaning of all ICB’s, replacement of all main board capacitors, replacement of all internal cooling units, ROM battery replacement, re-assembly, and mandatory final testing to make sure it meets our factory specifications. Turnaround time is approximately 3 weeks.
RMA 5686 has been assigned to Caltech’s returning TC010M controller. Attached please find our RMA forms. Complete and return them to us via email, along with your PO, prior to shipping the cont
Osaka Vacuum USA, Inc.
510-770-0100 x 109
our TP-1 TG390MCAB is 9 years old. What is the life expectancy of this turbo?
The Osaka maglev turbopumps are designed with a 100,000 hours(or ~ 10 operating years) life span but as you know most of our end-users are
running their Osaka maglev turbopumps in excess of 10+, 15+ years continuously. The 100,000 hours design value is based upon the AL material being rotated at
the given speed. But the design fudge factor have somehow elongated the practical life span.
We should have the cost of new maglev & controller in next year budget. I put the quote into the wiki.
Chub Osthelder received 40m specific basic safety traning today.
Steve reported to me that the CC1 Hornet gauge was not reporting the IFO pressure after some cable tracing at EX. I found that the power to the unit had been accidentally disconnected. I re-connected the power and manually turned on the HV on the CC gauge (perhaps this can be automated in the new vacuum paradigm). IFO pressure of 8e-6 torr is being reported now.
Physical plan is cleaning our roof and gutters today.
Let's install Jamie's new Data Viewer
I'm continuing the arm loss measurements Yuki was making. I'm first familiarizing myself with the procedures for the measurement Johannes describes.
I'm not very familiar with the medm screens, so I'm just kind of poking around and checking with Gautam. I do the following:
I've left the script running.
Some facts which should be considered when doing this measurement and the associated uncertainty:
After running this script Friday night, i noticed Saturday that the data hadn't saved. Scrolling up inthe terminal, I couldn't see where I'd run the script, so I thought I'd forgotten to run it as I was making last minute changes to the scope settings Friday before leaving.
Monday it turns out I hadn't forgotten to run the script, but the script itself was getting hung up as it waited for ASS to settle, due to the offset on the ETM PIT or YAW setpoints. The script was waiting until both pitch and yaw settled to below 0.7, but yaw was reading ~15; I think this is normal, and it looks like Yuki had solved this problem by waiting for the DEMOD-OFFSET to become small, rather than just the DEMOD signal to be small. Since this is a solved problem, I think I might be using an old script, but I'm pretty sure I'm running the one in Johannes' folder that Yuki is referencing for example here. The scripts in /yutaro_scripts/ have this DEMOD-OFFSET functionality commented out, and anyway those scripts seem to do the 2D loss maps rather than 1D loss measurements.
In the meantime I blocked the beams and ran the script in DARK mode. The script is saving data in /armloss/data/run_20181105/, and runs with no exceptions thrown.
However, when I try to dither align the YARM, I get an error that "this is not a degree of freedom that has an ASS". I'm alsogetting some exceptions from MEDM about unavailable channels. It must have been something about donatella not initializing, because it's working on pianosa. I turned on YARM ADS from pianosa. Monitoring from dataviewer, I see that LSC-TRY_OUT has some spikes to 0.5, but it's mostly staying near 0. I tried returning to the previous frozen outputs, and also stepping around ETMY-[PIT/YAW] from the IFO_ALIGN screen, but didn't see much change in the behavior of LSC-TRY. I missed the other controls Gautam was using to lock before, and I've also made myself unclear on whether ASS is acting only on angular dof, or also on length.
I unblocked the beams after the DARK run was done.
The Contec test board with Dsub37Fs was on the top shelf of E7
I'm checking out the data this morning, running armloss_AS_calc.py using the parameters Yuki used here.
I made the following changes to scripts (measurement script and calculator script)
I repeated the 'dark' measurements, because I need 20 files to run the script and the measurements before had the window on the scope set larger than the integration time in the script, so it was padded with bad values that were influencing the calculation.
On running the script again, I'm getting negative values for the loss. I removed the beamstops from the PDs, and re-centered the beams on the PDs to repeat the YARM measurements.
The IMC has been misbehaving for the last 5 hours. Why? I turned the WFS servos off. afaik, aaron was the last person to work on the IFO, so i'm not taking any further debugging steps so as to not disturb his setup.
I tried to plot a long trend MC Transmitted today. I could not get farther than 2017 Aug 4
The mode cleaner was misaligned probably due to the earthquake (the drop in the MC transmitted value slightly after utc 7:38:52 as seen in the second plot). The plots show PMC transmitted and MC sum signals from 10th june 07:10:08 UTC over a duration of 17 hrs. The PMC was realigned at about 4-4:15 pm today by rana. This can be seen in the first plot.
That was likely me. I had recentered the beam on the PD I'm using for the armloss measurements, and I probably moved the wrong steering mirror. The transmission from MC2 is sent to a steering mirror that directs it to the MC2 transmission QPD; the transmission from this steering mirror I direct to the armloss MC QPD (the second is what I was trying to adjust).
Note: The MC2 trans QPD goes out to a cable that is labelled MC2 op lev. This confusion should be fixed.
I realigned the MC and recentered the beam on the QPD. Indeed the beam on MC2 QPD was up and left, and the lock was lost pretty quickly, possibly because the beam wasn't centered. Lock was unstable for a while, and I rebooted C1PSL once during this process because the slow machine was unresponsive.
When tweaking the alignment near MC2, take care not to bump the table, as this also chang es the MC2 alignment.
Once the MC was stably locked, I was able to maximize MC transmission at ~15,400 counts. I then centered the spot on the MC2 trans QPD, and transmission dropped to ~14800 counts. After tweaking the alignment again, it was recovered to ~15,000 counts. Gautam then engaged the WFS servo and the beam was centered on MC2 trans QPD, transmission level dropped to ~14,900.
Jon and I stuck a extender card into the eurocrate at 1X8 earlier today (~5pm PT), to see if the box was getting +24V DC from the Sorensen or not. Upon sticking the card in, the FAIL LEDs on all the VME cards came on. We immediately removed the extender card. Without any intervention from us, after ~1 minute, the FAIL LEDs went off again. Judging by the main volume pressure (Attachment #1) and the Vacuum MEDM screen (Attachment #2), this did not create any issues and the c1vac1 computer is still responsive.
But Steve can perhaps run a check in the AM to confirm that this activity didn't break anything.
Is there a reason why extender cards shouldn't be stuck into eurocrates?
Please check your data file and compare with those Johannes made last year. I think the power in your data file may have only three-disits and flactuate about 2%, which brings huge error. (see elog: 40m/14254)
On running the script again, I'm getting negative values for the loss.
The vacuum and MC are OK
The VEA vertex laptop, paola, has a flashing orange indicator which I take to mean some kind of battery issue. When the laptop is disconnected from its AC power adaptor, it immediately shuts down. So this machine is kind of useless for its intended purpose of being a portable computer we can work at optical tables with. The actual battery diagnostics (using upower) don't report any errors.
Earlier today, I rebooted a few unresponsive VME crates (susaux, auxey).
The IMC has been unhappy for a couple of days - the glitches in the MC suspensions are more frequent. I reset the dark offsets, minimized MCREFL by hand, and then re-centered the beam on the MC2 Trans QPD. In this config, the IMC has been relatively stable today, although judging by the control room StripTool WFS control signal traces, the suspension glitches are still happening. Since we have to fix the attenuator issue anyways soon, we can do a touch-up on IMC WFS.
I removed the DC PD used for loss measurements. I found that the AS beam path was disturbed - there is a need to change the alignment, this just makes it more work to get back to IFO locking as I have to check alignment onto the AS55 and AS110 PDs.
Single arm locking worked with minimal effort - although the X arm dither alignment doesn't do the intended job of maximizing the transmission. Needs a checkup.
PRMI locking (carrier resonant) was also pretty easy. Stability of the lock is good, locks hold for ~20 minutes at a time and only broke because I was mucking around. However, when the carrier is resonant, I notice a smeared scatter pattern on the ITMX camera that I don't remember from before. I wonder if the FF idea can be tested in the simpler PRMI config.
After recovering these two simpler IFO configurations, I improved the cavity alignment by hand and with the ASS servos that work. Then I re-centered all the Oplev beams onto their respective QPDs and saved the alignment offsets. I briefly attemped DRMI locking, but had little success, I'm going to try a little later in the evening, so I'm leaving the IFO with the DRMI flashing about, LSC mode off.
I had some success today. I hope that the tweaks I made will allow working with the DRMI during the day as well, though it looks like the main limiting factor in lock duty cycle is angular stability of the PRC.
[Attachment #1]: Repeatable and reliable DRMI locks tonight, stability is mainly limited by angular glitches - I'm not sure yet if these are due to a suspect Oplev servo on the PRM, or if they're because of the tip-tilt PR2/PR3/SR2/SR3.
[Attachment #2]: A pass at measuring the TF from SRCL error point to MICH error point via control noise re-injection. I was trying to measure down to 40 Hz, but lost the lock, and am calling it for the night.
[Attachment #3]: Coherence between PRM oplev error point and beam spot motion on POP QPD.
Note that the MICH actuation is not necessarily optimally de-coupled by actuating on the PRM and SRM yet (i.e. the latter two elements of the LSC output matrix are not precisely tuned yet).
What is the correct way to make feedforward filters for this application? Swept-sine transfer function measurement? Or drive broadband noise at the SRCL error point and then do time-domain Wiener filter construction using SRCL error as the witness and MICH error as the target? Or some other technique? Does this even count as "feedforward" since the sensor is not truly "outside" the loop?
This problem resurfaced. I'm doing the debugging.
6:30pm - "Solved" using the same procedure of stepping through the whitening gains with a small (10 DAC cts pk) signal applied. Simply stepping through the gains with input grounded doesn't seem to do the trick.
With the DRMI locked, I drove a line in MICH using the sensing matrix infrastructure. Then I looked at the error points of MICH, PRCL and SRCL. Initially, the sensing line oscillator output matrix for MICH was set to drive only the BS. Subsequently, I changed the --> PRM and --> SRM matrix elements until the line height in the PRCL and SRCL error signals was minimized (i.e. the change to PRCL and SRCL due to the BS moving, which is a geometric effect, is cancelled by applying the opposite actuation to the PRM/SRM respectively. Then I transferred these to the LSC output matrix (old numbers in brackets).
MICH--> PRM = -0.335 (-0.2655)
MICH--> SRM = -0.35 (+0.25)
I then measured the loop TFs - all 3 loops had UGFs around 100 Hz, coinciding with the peaks of the phase bubbles. I also ran some sensing lines and did a sensing matrix measurement, Attachment #1 - looks similar to what I have obtained in the past, although the relative angles between the DoFs makes no sense to me. I guess the AS55 demod phase can be tuned up a bit.
The demodulation was done offline - I mixed the time series of the actuator and sensor signals with a "local oscillator" cosine wave - but instead of using the entire 5 minute time series and low-passing the mixer output, I divvied up the data into 5 second chunks, windowed with a Tukey window, and have plotted the mean value of the resulting mixer output.
Unrelated to this work: I re-aligned the PMC on the PSL table, mostly in Pitch.
Gautam was doing some DRMI locking, so I replaced the photodiode at the AS port to begin loss measurements again.
I increased the resolution on the scope by selecting Average (512) mode. I was a bit confused by this, since Yuki was correct that I had only 4 digits recorded over ethernet, which made me think this was an i/o setting. However the sample acquisition setting was the only thing I could find on the tektronix scope or in its manual about improving vertical resolution. This didn't change the saved file, but I found the more extensive programming manual for the scope, which confirms that using average mode does increase the resolution... from 9 to 14 bits! I'm not even getting that many.
There's another setting for DATa:WIDth, that is the number of bytes per data point transferred from the scope.
I tried using the *.25 scope instead, no better results. Changing the vertical resolution directly doesn't change this either. I've also tried changing most of the ethernet settings. I don't think it's something on the scripts side, because I'm using the same scripts that apparently generated the most recent of Johannes' and Yuki's files; I did look through for eg tds3014b.py, and didn't see the resolution explicitly set. Indeed, I get 7 bits of resolution as that function specifies, but most of them aren't filled by the scope. This makes me think the problem is on the scope settings.
sstop using the ssscope, and just put the ssssignal into the DAQ with sssssome whitening. You'll get 16 bitsśšß.
I've been looking into the cross-coupling from the SRCL loop control point to the Michelson error point.
[Attachment #1] - Swept sine measurement of transfer function from SRCL_OUT_DQ to MICH_IN1_DQ. Details below.
[Attachment #2] - Attempt to measure time variation of coupling from SRCL control point to MICH error point. Details below.
[Attachment #3] - Histogram of the data in Attachment #2.
[Attachment #4] - Spectrogram of the duration in which data in #2 and #3 were collected, to investigate the occurrance of fast glitches.
Hypothesis: (so that people can correct me where I'm wrong - 40m tests are on DRMI so "MICH" in this discussion would be "DARM" when considering the sites)
Measurement details and next steps:
Attachments #2 and #3
This problem resurfaced, which I noticed when I couldn't get the single arm locks going.
The fix was NOT restarting the c1rfm model, which just brought the misery of all vertex FEs crashing and the usual dance to get everything back.
Restarting the sender models (i.e. c1scx and c1scy) seems to have done the trick though.
It is posted at the 40m wiki with Gautam' help. Printed copies posted around doors also.