We setup the channels for PID control of the seismometer can. First, we ssh into c1auxex and went to /cvs/cds/caltech/target/c1auxex2 and found ETMXaux.db. We then added in new soft channels that we named C1:PEM-SEIS_EX_TEMP_SLOWKP, C1:PEM-SEIS_EX_TEMP_SLOWKI, C1:PEM-SEIS_EX_TEMP_SLOWKD that will control the proportional, integral and differential gain respectively. These channels are used in the script FSSSlow.py for PID control. We then had to restart the system, but first we turned off the LSC mode and then shut down the watchdog on the X end. After doing the restart, we disabled the OPLEV as well before restarting the watchdog. Then, we enabled the LSC mode again. This is done to not damage any of the optics during the restart. The restart is done by using sudo systemctl restart modbusIOC.service and restarted with sudo systemctl status modbusIOC.service. Then, we made sure that the channels existed and could be read and writtten to, so we tried z read [channel name] and it read 0.0. We then did z write [channel name] 5, and it wrote 5 to that channel. Now that the channels work, we can implement the PID script and check to make sure that it works as well.
CC1 old MKS cold cathode gauge randomly turns on- off. This makes software interlock close VM1 to protect RGA So the closed off RGA region pressure goes up and the result is distorted RGA scan.
CC1 MKS gauge is disconnected and VM1 opened. This reminds me that we should connect our interlocks to CC1 Hornet Pressure gauge.
Pumpdown 80 at 511 days and pd80b at 218 days
Valve configuration: special vacuum normal, annuloses are not pumped at 3 Torr, IFO pressure 7.4e-6 Torr at vac envelope temp 22 +- 1C degrres
While Kevin and Arijit were doing their MC_REFL PD noise measurements (which they will elog about separately shortly), I noticed a feature around 600kHz that reminded me of the NPRO noise eater feature. This is supposed to suppress the relaxation oscillation induced peak in the RIN of the PSL. Surprisingly, the noise eater switch on the NPRO front panel was set to "OFF". Is this the normal operating state? I thought we want the noise eater to be "ON"? Have to measure the RIN on the PSL table itself with one of the many available pick off PDs. In any case, as Attachment #1 showed, turning the noise eater back on did not improve the excess IMC frequency noise.
Todd informed me that the ADC Timing adaptor boards we had ordered arrived today. I had to solder on the components and connectors as per the schematic, though the main labor was in soldering the high density connectors. I then proceeded to shut down all models on c1lsc (and then the FE itself). Then classic problem of all vertex machines crashing when unloading models on c1lsc happened (actually Koji noticed that this was happening even on c1ioo). Anyways this was nothing new so I decided to push ahead.
I had to get a cable from Downs that connects the actual GS ADC card to this adaptor board. I powered off the expansion chassis, installed the adaptor board, connected it to the ADC card and restarted the expansion chassis and also the FE. I also reconnected the SCSI cable from the AA board to the adaptor card. It was a bit of a struggle to get all the models back up and running again, but everything eventually came back(after a few rounds of hard rebooting). I then edited the c1x04 and c1lsc simulink models to reflect the new path for the X arm ALS error signals. Seems to work alright.
At some point in the afternoon, I noticed a burning smell concentrated near the PSL table. Koji traced the smell down to the c1lsc expansion chassis. We immediately powered the chassis off. But Steve later informed me that he had already noticed an odd burning smell in the morning, before I had done any work at the LSC rack. Looking at the newly installed adaptor card, there wasn't any visual evidence of burning. So I decided to push ahead and try and reboot all models. Everything came back up normally eventually, see Attachment #1. Particle count in the lab seems a little higher than usual (actually, according to my midnight measurement, they are ~factor of 10 lower than Steve's 8am measurements), but Steve didn't seem to think we should read too much into this. Let's monitor the situation over the coming days, Steve should comment on the large variance seen in the particle counter output which seems to span 2 orders of magnitude depending on the time of the day the measurement is made... Also note that there is a BIO card in the C1LSC expansion chassis that is powered by a lab power supply unit. It draws 0 current, even though the label on it says otherwise. I a not sure if the observed current draw is in line with expectations.
The spare (unstuffed) adaptor cards we ordered, along with the necessary hardware to stuff them, are in the Digital FE hardware cabinet along the east arm.
Steve: particle count in the 40m is following outside count, wind direction, weather condition .....etc. The lab particle count is NOT logged ! This is bad practice.
we measured the RIN of the MC2 transmission using the PDA255 I had put on the MC2 trans table sometime ago for ringdowns. Attached are (i) spectra for the RIN, (ii) spectra for the classical rad. pressure noise assuming 500W circulating power and (iii) a tarball of data and code used to generate these plots.
We took a full span measurement (to make sure there aren't any funky high-freq features) and a measurement from DC-800 Hz (where we are looking for excess noise). The DC level of light on the photodiode was 2.76V (measured using o'scope)
I'll add this to the noise budget later. But the measured RIN seems consistent with a 2013 measurement at 100Hz (though the 2013 measurement is using DTT and so doesn't have high frequency information).
While at the MC2 table, we noticed that it has some optical problems:
We estimated that the power in the IMC is (1 W)*Finesse/pi = 500 W. The MC2 Transmission spec is < 10 ppm, so the power on the table is probably ~5 mW. Since the PDA255 has a transimpedance of 10 kOhm and a max output power of 10V, it can handle up to ~1 mW. Probably we can get the QPD to handle 4 mW.
Gautam, Steve 3-27
We measured MC2 transmitted power right at the uncoated window ~2.5 mW The beam was just a little bigger than the meter.
I made sketches of the final setup. There will be a box in the rack that contains both the heater circuit and the temperature sensor boards. One of them is in the loop while the other isn't. Instead of having many cables leading to the can, there will only be these three, though they can be made into a single wire. It will be connected to the can through a D-9 connector. The second attachment is what will be inside of the box, with all the major wires and components labeled.
Edit: I've canged the layout to (hopefully) make the labels easier to read. I've also added in a cable to the ADC that reads out the voltage across the 1 ohm resistor. I also attached the circuit diagrams for the heater circuit and the temperature sensors. The one for the heater circuit was made by Kevin and I used the same design, except I have LM7818 and LM7918, since the 15V ones were not available at the time I made the circuit.
In addition, all the wires leading to the can will all be part of one bundle of wires (I didn't clearly indicate it as such). There will be a total of 6 wires: two are needed for the wire to supply power to the heater and will have a LEMO connector on the rack end and two are needed for each temperature sensor, which will be attached to the board directly on the rack end.
Also, we don't need two voltage regulators for each temperature circuit. We can just have one of each of LM7815 and LM7915 to supply +/- 15V to the boards.
PMC and IMC re-aligned and re-locked. Both cavities are staying locked. Arm cavities are also locked.
I fit the data that we got from the test. The time constant for the cooling came out to be about 4.5 hours. The error is quite large and we should add a low pass filter to the temperature sensor eventually in order to minimize the noise of the measurements.
we don't ever want to use our 16 kHz real time system for such low frequency action; its main purpose is for real-time controls, whereas we are OK with multiple seconds of delay in a thermal loop. The Python PID script is sufficient and highly reliable (after years of testing).
I've added two curves to the NB. Both are measured (with FET preamp) at the output of the demod board, with the LO driven at the nominal level by the Wenzel RF source pickoff (as it would be when the IMC is locked) and the RF input connected to the IMC REFL PD. For one curve, I simply closed the PSL shutter, while for the other, I left the PSL shutter open, but macroscopically misaligned MC2 so that there was no IMC cavity. So barring RFAM, there should be no PDH signal on the REFL PD, but I wanted to have light on there. I'm not sure if I understand the difference between these two curves though, need to think on it. Perhaps the IMC REFL PD's optical/electrical response needs to be characterized?
Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
I did a preliminary noise budget of the transmitted frequency noise of the IMC. Attachment #1 shows the NB. I'm going to use this opportunity to revisit my IMC modeling. Some notes:
Conclusion: From this study, assuming my PDH discriminant calibration was correct, looks like IMC demod / POX11 demod electronics noises are not to blame (this surprises me since there were apparently so many things wrong on the demod board, and yet that wasn't the worst thing in the IMC chain it would seem ). The POX11 photodiode "dark" noise is also not the problem I think, given the grey curve. Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
Using this, I can now make up a noise budget of sorts for the IMC sensing.
As discussed at the meeting, I decided to calibrate the MC error point into physical units of Hz/rtHz (a.k.a. the PDH discriminant). This is to facilitate the debugging of the hypothesized excess IMC sensing noise. I did this as follows.
gautam 20180327 4.30pm: I re-checked the PDH error signal calibration using the oscilloscope method. Attachment #3 shows the PDH I and Q error signals and also the output of the RF monitor port, during a TEM00 flash. This attachment should be compared to Attachment #2 of elog 12822, and the answer lines up quite well. From my Finesse model of the IMC, I calculated that the x-axis of the PDH horn-to-horn is ~12.3kHz. Comparing to the top row of Attachment #3, I get a PDH error signal calibration of ~12.4kHz/Vrms, which lines up well with the number quoted above. So I trust my calibration, and hence, the y-axis of my noise budgets in reply to this elog.
Light bulb replaced.
Bulb went out ~10am today. Looks like the lifetime of this bulb was <100 days.
Steve: bulb is arriving next week
After some persistence, I managed to get the mixers off.
Unfortunately, the coherent noise between the arms persists so the sensing noise injection must be happening elsewhere. IMC seems to lock fine though so I'm leving the autolocker on
This elog by koji inspired me to consider power supply as a possible issue.
The demod board receives +/-24V DC (which is regulated down to +/-15V DC by 7815/7915), and also +15V DC via the backplane. The ERA-5SM receives DC power from the latter (unregulated) +15V DC. I can't think of why this is the case except perhaps the regulators can't source the current the amp wants? In any case, it doesn't look feasible to change this by cutting any traces on the PCB to me. While I had the board out, I decided to replace the JMS-1H mixers in a last ditch effort to improve the demod board noise. Unfortunately I'm having trouble de-soldering these MCL components from the board. So for now, I'm leaving the demod board out, IMC unlocked. Work will continue tomorrow.
according to the temp sensor readout, which was ~-3.35V which corresponds to ~335K, the temperature of the can is now 60 deg C. This is a bit warm for my liking so i'm turning the heater current down to 0 now by writing 0 to C1:PEM-SEIS_EX_TEMP_CTRL
The MOSFET was getting pretty hot, so I switched it out to a larger heat sink and it's fine now. I then used a function generator in place of the DAC to provide ~3.5V. I got the current in the circuit to 1.7A, which is as expected, since we have 24V input, the heater resistance is 12.5ohm and the resistor we are using is 1ohm, so 24V/(12.5+1)ohm = 1.7A. The temperature inside the can rose about 5 degrees in half an hour. The only issue now is the voltage regulators and OP amp inside the box get hot, though it doesn't seem to be dangerous. I switched the function generator input to a DAC and Gautam set it to 1.5V. If it works, then we'll leave this on overnight and work on the PID control tomorrow. I've attached images of the current heater circuit box when it is open and the new heat sink for the MOSFET.
gautam: we also tried incorporating the EPICS channels from the Acromag into the RTCDS so that we can implement PID control by using Foton. I tried doing this using the "EpicsIn" and "EpicsOut" blocks from CDS_PARTS. While the model recompiled smoothly, I saw no signals in the filter module i had connected in series with the EpicsIn block. So I just reverted c1pem to its original state and recompiled the model. Guess we will stick to python script PID reading EPICS channels to implement the PID servo.
Re-measured the demod board noises after replacing the suspect ERA-5SMs, with LO driven by a marconi at the "nominal" level of 2.5dBm, and RF input terminated. Attachment #1 is the input referred voltage noise spectra. I used the FET low noise pre-amp box for this purpose. I cannot explain the shape of the spectra above 1kHz. I tried doing the measurement on a minicircuits mixer (non-surface mount) and found the shape to be flat throughout the SR785 span. Unclear what else could be going on in the demod board though, all the other components on it are passive (except the ERA-5SMs which were replaced). I considered adopting a PMC style demod setup where we do the demod using some separate Minicircuits Mixer+LowPass filter combo. But the RF flashes for the IMC monitored at the RFmon port are ~0.2Vpp, and so the RF input to the mixer is expected to be ~2Vpp. The minicircuits mixer selection guide recommends choosing a diode mixer with LO level at least 10dBm above the expected RF input signal level, and we don't have any standalone mixers that are >Level 7. I've asked Steve to package the aLIGO demod board in the meantime, but even that might not be a plug and play replacement as the IF preamp stage has ~120degrees phase lag at 1MHz, which is significantly higher than the existing board which just has a SCLF5 low pass filter after the mixer and hence has <45degrees phase lag at 1MHz.
It is not clear to me why installing an attenuator to prevent amplifier saturation has necessitated a 10dB increase in the IN1 gain and 3dB increase in the VCO gain. Initially, I was trying to compensate for the gain by increasing the FSS "Common Gain" but in that setting, I found an OLTF measurement impossible. The moment I enabled the excitation input to the CM board, the lock was blown, even with excitation amplitudes as small as -60dBm (from the Agilent network analyzer).
This may also be a good opportunity to test out one of the aLIGO style FET mixer demod boards (recall we have 2 spare from the 4 that were inside the ALS demod box). I'm going to ask Steve to package these into a 1U chassis so that I can try that setup out sometime. From a noise point of view, the aLIGO boards have the advantage of having a x100 preamp stage straight after the mixer+LPF. We may need to replace the lowpass filter though, I'm not sure if the one installed is 1.9MHz or 5MHz.
I've left an SR785 and AG4395 near 1X2 in anticipation of continuing this work tomorrow.
Unrelated to this work - seems like the WFS DC and RF offsets had not been set in a while so I reset these yesterday. The frequent model restarts in recent times may mean that we have to reset these to avoid using dated offset values.
The working hypothesis, since the excess noise in single arm locks is coherent between both arms, the excess sensing noise is frequency noise in the IMC locking loop (sensing because it doesn't show up in MC_F). I've started investigating the IMC sensing chain, starting with the power levels of the RF modulation source. Recall that we had changed the way the 29.5MHz signal was sent to the EOM and demod electronics in 2017. With the handheld RF power meter, I measured 13.2dBm coming out of the RF distribution box (this is routed straight from the Wenzel oscillator). This is amplified to 26dBm by an RF amplifier (ZHL-2-S) and sent to the EOM, with a coupled 16dBm part sent to a splitter that supplies the LO signal to the demod board and also the WFS boards. Lydia made a summary of expected RF power levels here, and I too seem to have labelled the "nominal" LO level to the MC_REFL demod board as +5dBm. But I measured 2.7dBm with the RF power meter. But looking closely at the schematic of the splitting circuitry, I think for a (measured) 16.7dBm input to it, we should in fact expect around 3dBm of output signal. So I don't know why I labelled the "nominal" signal level as 5dBm.
Bottom line: we are driving a level 17 mixer with more like +14dBm (a number inferred from this marked up schematic) of LO, which while isn't great, is unlikely to explain the excess noise I think (the conversion loss just degrades by ~1dB). So I will proceed to check further downstream in the signal chain.
The temperature control output channel for the XEND seismometer wasn't working properly. The EPICS channel existed, could be written to and read from, but no physical voltage was observed on the (confirmed properly) wired connector.
The Acromag DAC that outputs this channel was completely spare in the original scheme and does not serve any other channels at the moment. We found it to be unresponsive to ping from the host machine (reminder: the Acromags are on their own subnet with IPs 192.168.114.xxx connected to the secondary ethernet adapter of c1auxex), while all others returned the ping just fine. The modules have daisy-chained ethernet connections, and the one Acromag unit behind the unresponsive one in the chain was still responding to ping and its channels were working, so it couldn't have been a problem with the (ethernet) cabling.
Gautam and I power-cycled the chassis and server, which resolved the issue. The channel is now outputting the requested voltage on the Out1 BNC connector of the chassis (front). When I was setting up the whole system and did frequent rebooting and IP-redefinitions I have seen network issues arise between server and Acromags. In particular, when changing the network settings server-side, the Acromags needed to reboot occasionally. So this whole problem was probably due to the recent server-swap, as the chassis had not been power-cycled since.
During the debugging we also found that the c1psl2 channels were not working. This was because I had overlooked to update the epics environment variables for the modbus path defined in /cvs/cds/caltech/target/c1psl2/npro_config.cmd from the local installation /opt/epics/ (which doesn't exist on the new server anymore) to the network location /cvs/cds/rtapps/epics-220.127.116.11_long/. This has been fixed and the slow diagnostic PSL channels are recording again.
I am currently working on an optical arrangement consisting of a QPD that measures the fluctuations of an incoming HeNe laser beam that is reflected by a mirror. The goal is to add a second QPD to the optical arrangement to form a linear combination that effectively cancels out the (angular) fluctuations from the laser beam itself so that we can only focus on the fluctuations produced by the mirror.
In order to solve this problem, I have written a program for calculating the different contributions of the fluctuations of the HeNe laser and fluctuations from the mirror, for each QPD (program script attached). The goal of the program is to find the optimal combination of L0, L1, L2, and f2 that cancels the fluctuations from the laser beam (while retaining solely the fluctuations from the mirror) when adding the fluctuations of QPD 1 and QPD 2 together.
By running this program for different combinations of distances and focal lengths, I have found that the following values should work to cancel out the effects of the oscillations from the HeNe laser beam (assuming a focal length of 0.2 m for the lens in front of the original QPD):
L0 = 1.0000 m (distance from laser tube to mirror)
L1 = 0.8510 m (distance from mirror to lens in front of QPD 1)
L2 = 0.9319 m (distance from beamsplitter to lens in front of QPD 2)
f2 = 0.3011 m (focal length of lens in front of QPD 2)
Based on these calculations, I propose to try the following lens for QPD 2:
1’’ UV Fused Silica Plano-Convex Lens, AR-Coated: 350 - 700 nm (focal length 0.3011 m). https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=6508
Valve configuration: special vacuum normal, annuloses are not pumped at 3 Torr, IFO pressure 7.4e-6 Torr at vac envelope temp 22 +- 1C degrees
pd80b rga scan at 175 day. IFO pressure 7.3e-6 Torr-it
Condition: vacuum normal, annuloses not pumped. Rga turned on yesterday.
The rga was not on since last poweroutage Jan 2, 2018 It is warming up and outgassing Atm2
I have attached the setup I completed today. The metal box contains the heater circuit and the board for the temperature sensor is right above it. This is basically the same setup as before, but I've just packaged everything up neater. I expect to be able to perform the test tomorrow and begin implementing PID control. I still need a DAC input for the heater circuit and the temperature sensor is having some issues as well.
The cert for nodus has been renewed for another 2 years.
The following is the basic procedure for getting a new cert: (Note certs are only good for two years as of 2018)
openssl req -sha256 -nodes -newkey rsa:2048 -keyout nodus.ligo.caltech.edu.key -out nodus.ligo.caltech.edu.csr
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:CaliforniaLocality Name (eg, city) :Pasadena
Organization Name (eg, company) [Internet Widgits Pty Ltd]:California Institute of Technology
Organizational Unit Name (eg, section) :LIGO
Common Name (eg, YOUR name) :nodus.ligo.caltech.edu
Leave the e-mail address, challenge password and optional company name blank. A new private key will be generated.
chown root nodus.ligo.caltech.edu.key
chgrp root nodus.ligo.caltech.edu.key
chmod 0600 nodus.ligo.caltech.edu.key
The nodus.ligo.caltech.edu.csr file is what is sent in for the cert.
This file should be sent to either email@example.com or firstname.lastname@example.org and copy email@example.com.
A URL llink with the new cert to be downloaded will be sent to the requestor.
Once the files are downloaded, the new cert and intermediate cert, they can be copied and renamed.
The PEM-encoded host certificate by itself is saved at:
The nodus.ligo.caltech.edu.key file should be in the same directory or whichever directory is indicated in the ssl.conf located in /etc/httpd/conf.d/ directory.
httpd will need to be restarted in order for it to see the new cert.
I replaced the borrowed server with the permanent one today. Before Removing the current server, Before, I performed several additional preparations:
apt-get install procserv
Then I replaced the server:
I returned c1auxex2 to Larry, who needed it back asap because of some hardware failure
Steve: Acromag XT1221 ordered 3-15-18
I assembled the rack-mount server that will long-term replace c1auxex, so we can return the borrowed unit to Larry.
SUPERMICRO SYS-5017A-EP Specs:
I installed a standard Debian Jessie distribution, with option LXDE for minimal resource usage. Steps taken after fresh install
iface eth0 inet static
dns-nameservers 192.168.113.104 18.104.22.168 22.214.171.124
apt-get install libmotif-common libmotif4 libxp6 (
required to run burtwb utility)
The server is ready to take over for c1auxex2 and does not need any local epics compiled, since it can run the 126.96.36.199_long binaries in /cvs/cds.
Kevin suggested I shouldn't be so lazy and test the POY spectrum as well. So we moved the timing card back to c1iscey, went through the usual dance of vertex machine reboots, and then got both single arm locks going. Attached spectrum shows that both POX and POY are noisy. I'm not sure what has changed that could cause this effect. The fact that both POX and POY appear uniformly bad, but that there is no coherence with MC_F, suggests to me that perhaps this has something to do with the work I did with Koji w.r.t. the power situation at the LSC rack. But we just checked that
Another observation we made: note the huge bump around 70Hz in both arm control signals. We don't know what the cause of this is. But we occassionally noticed harmonics of this (i.e. 140, 210 Hz etc) appear in the control signal spectra, and they would grow with time - eventually, the X arm would lose lock (though the Y arm stayed locked).
I'm short on ideas for now so we will continue debugging tomorrow.
Unrelated to this work: Kevin reminded me that the high-pitched whine from the CRT TVs in the control room (which is apparently due to the flyback transformer) is DEAFENING. It's curious that the "chirp" to the eventual 15kHz whine is in opposite directions for the QUAD CRTs and the single display ones. Should be a Ph6 experiment maybe.
Update 2:30pm Mar 13: The furthest back I seem to be able to go in time with Frames is ~Jan 20 2018. Looking for a time when the arms were locked from back then, it seems like whatever is responsible for a noisy POX and POY was already a problem back in January. See Attachment #2. So it appears that the recent work at 1Y2 is not to blame...
we tested my noisy POX hypothesis tonight. By locking the single arm with POX, the arm length is forced to follow PSL frequency, which is itself slaved to IMC length. From Attachment #1, there is no coherence between the arm control signal and MC_F. This suggests to me that the excess noise I am seeing in the arm control signal above 30 Hz is not originating from the PSL. It also seems unlikely that at >30Hz, anything mechanical is to blame. So I am sticking with the hypothesis that something is wonky with POX. For reference, a known "normal" arm control signal spectrum looks like the red curve in this elog.
Bulb is replaced.
1. Optical Table Layout
I had discussed with Koji a way to record coordinates of optical table equipments in a text file, and load to solidworks. The goal is to make it easier to move things around on the table in the CAD. While I have succeeded in importing coordinates through txt files, there is still a lot of tediousness in converting these points into sketches. Furthermore, the task has to be redone everytime a coordinate is added to or changed in the txt file. Koji and I think that this can all be automated through solidworks macros, so I will explore that option for the next two weeks.
2. Vacuum Chamber CADs
Steve will help find manufacturing drawings of the BS chamber. I have completed the ETM chambers, while the ITM ones are identical to them so I will reuse parts for the CAD.
I ceated a simple circuit that takes in 15V and outputs precisely 5V by using a 12V voltage regulator LM7812 and an AD586 that takes the output of the voltage regulator and outputs 5V (attachment 1). We plugged this into the slow channel and will leave it running for a few hours to see if we still have the fluctuations we observed earlier and also fit the noise curve. We'll also test the fast channel later as well. Attachment 2 shows the setup we have in the lab, with the red and white cable plugged into the +15V power supply and the red and black cable connected to the slow channel.
I was going to head out but then it occurred to me that I could do another simple test, which is to try and lock the X arm on ALS error signal (i.e. actuate on MC length to keep the beat between EX laser and PSL fixed, while the EX frequency is following the Xarm length). Comparing the in loop (i.e. ALS) error signal with the out-of-loop sensor (i.e. POX), it seems like POX is noisy. The curves were lined up by eye, by scaling the blue curve to match the red at the ~16Hz peaks. This supports my hypothesis in the previous elog. On the downside, could be anything. Electronics in the POX chain? The demod unit itself? Will look into it more tomorrow..
As an aside, controlling the arm with ALS error signal worked quite well, and the lock was maintained for ~1 hour.
Then I looked at the spectrum, see Attachment #1. Disappointingly, it looks like the arm PDH servo is dominating the noise, and NOT unsuppressed EX laser frequency noise,. Not sure why this is so, and I'm feeling too tired to debug this tonight. But encouragingly, the performance of the new ALS signal chain looks very promising. Once I tune up the X arm loop, I'm confident that the ALS noise will be at least as good as the reference trace.
I am leaving c1iscey shutdown until this is fixed. So ETMY is not available for the moment.
Random factoid: Trying to print a DTT trace with LaTeX in the label text on pianosa causes the DTT window to completely crash - so if you dont save the .xml file, you lose your measurement.
I made a LISO fit of the measured TF of the daughter board, so that I can digitally invert the daughter board whitening. Results attached. (Inverse) Filters have been uploaded to the ALS X Foton filter banks.
I was forced into a simultaneous power-cycle rebooting of the three vertex FEs just now. I took the opportunity to completely disconnect the c1sus expansion chassis from all power and then restart it.
Everything is back up right now, and the weird timing issues I noticed in the sus model seem to be gone now (I'll need a longer baseline to be sure and I'll post a trend of the CPU timing tomorrow). It's disconcerting that apparently the only way to get everything back up and running is the nuclear option of power-cycling all FE related electronics. I was considering borrowing an ADC adapter card from the Y end and measuring the calibrated IR ALS noise with the digital system, but if I'm going to have to go through this whole dance each time I do a model recompile on c1lsc (which I'm going to have to in order to get the extra ADC recognized), I'm wondering if it's just better to wait till we get the new adapter cards we ordered. I think I'm going to work on tuning the input coupling into the fiber at EX in the next couple of days instead.
Seems like some high level voodoo .
Yesterday, we installed some new DIN rail connectors at the LSC rack to provide 3 new outputs each for +24V DC and -24V DC. The main motivation was to facilitate the installation and powering of the differential receiving AA board. The regulators used inside the 1U chassis actually claims a dropout voltage of 0.5V and outputs 14V nominally, so a +/-15V DC supply would've perhaps been sufficient, but we decided to leave a bit more margin, and unfortunately, there are no +/-18V DC KEPCO linear power supplies to the LSC rack. Procedure:
The c1lsc frontend models crashed for some reason during this procedure. Now the c1sus frontend model is also behaving weirdly. It is unclear to me if/how this work would have led to these problems, but the temporal correlation (but not causation?) is undeniable.
As I had found before, restarting the c1oaf model fixed the DC error. There is however still a pesky red indicator light on the "ADC0" in c1oaf. Trying to open up the ADC MEDM screen to investigate this further leads to the blank screen on the bottom right of Attachment #1. Probably has something to do with the fact that the model has an ADC block (because every model needs one?) but no signals are actually being piped to the model directly from the ADC.
Another observation, though I don't have any hypothesis as to why this was happening: on the c1sus machine, the c1sus model would frequently overclock, and then eventually, crash. I observed this behaviour at least 3 times between last night and now. The other models seemed fine though, in fact, IMC stayed locked. Why should this have been the case? It remains to be seen if this was somehow connected to the red DC indicator on c1oaf, though why should this be the case? Isn't the DC just concerned with writing data to frames? Any sort of IPC should be independent? Attachment #2 shows that there's been a definite increase in the maximum time on c1sus clock-cycle since yesterday (it's a 10 day minute trend plot of the model clock cycle timing and also the maximum time). Why? Koji and I did switch off all the Sorensens at the LSC rack for about 30mins, but why should this affect anything at 1X6? There are no red lights in either the c1lsc or c1sus expansion chassis. Curiously, the PRM also seems to be glitchy - as I'm sitting in the control room, I see a spot flashing across vertically on the REFL CRT monitor sporadically. Note that nominally, with PRM misaligned, the REFL CRT should be dark. dmesg on c1sus doesn't shed any light on the issue.
This required multiple hard reboots, but seems like all the RT models are back for now. The only indicator I can't explain is the red DC field on c1oaf. Also, the SUS model seems to be overclocking more frequently than usual, though I can't be sure. The "timing" field of this model's state word is RED, while the other models all seem fine. Not sure what could be going on.
Will debug further tomorrow, when I probably will have to do all this again as I'll need to recompile c1lsc for the ALS electronics test with the new ADC card from the differential AA board.
I am almost ready for a digital test of the new ALS electronics. Today, Koji and I spent some time tapping new +/-24VDC DIN terminal blocks at the LSC rack to facilitate the installation of the 1U differential receiving AA chassis (separate elog entry). The missing piece of the puzzle now is the timing adapter card. I opted against trying a test tonight as I am having some trouble bringing c1lsc back online.
Incidentally, a repeat of the voltage noise measurement of the X arm ALS beat looked much cleaner today, see Attachment #1 - I don't have a good hypothesis as to why sometimes the signal has several harmonics at 10Hz multiples, and sometimes it looks just as expected. The problem may be more systematically debuggable once the signals are being digitally acquired.
Here are the plots. Comments:
I like LTspice for such modeling - the GUI is nice to have (though I personally think that typing out a nodal file a la LISO is faster), and compared to LISO, I think that the LTspice infrastructure is a bit more versatile in terms of effects that can be modeled. We can also easily download SPICE models for OpAmps from manufacturers and simply add them to the library, rather than manually type out parameters in opamp.lib for LISO. But the version available for Mac is somewhat pared down in terms of the UI, so I had to struggle a bit to find the correct syntax for the various simulation commands. The format of the exported data is also not as amenable to python plotting as LISO output files, but i'm nitpicking...
I've gotten the LTSpice model working now, will post the comparison of modelled output noise for various combinations here shortly.
I did a quick test of the noise of the new ALS electronics with the X arm ALS. Attachment #1 shows the results - but something looks off in the measurement, especially the "LO driven, RF terminated" trace. I will have to defer further testing to tomorrow. Of course the real test is to digitize these signals and look at the spectrum of the phase tracker output, but I wanted a voltage noise comparison first. Also, note that I have NOT undone the whitening TFs of (z,p) = (15,150) on these traces. I wonder if these noisy signals (particularly the 10Hz multiple harmonics) are an artefact of measurement, or if something is wonky in the daughter board circuits themselves. I am measuring these with the help of a DB9 breakout board and some pomona minigrabbers. Reagrdless, the sort of ripple seen in the olive green trace for the I channel wasn't present when I did the same test with RF signal generators out on the electronics workbench, so I am inclined to think that this isn't a problem with the circuit. I'm measuring with the SR785 with the "A" input setting, but with the ground set to "Float". I need to look into what the difference is between this mode, and the "A-B" mode. At first glance, both seem to be equivalent differential measurements, but I wonder if there is some subtlety w.r.t. pickup noise.
Perhaps I can repeat the test at the output of the AA board. I looked into whether there is a spare +/- 24V DC power supply available at the LSC rack, to power the 1U AA chassis, but didn't see anything there.
For testing the new IR ALS noise, we had decided that we would like to use the differential output of the demodulated ALS beat signal, as opposed to a single-ended output, as measurements suggested the former to be a lower noise configuration than the latter. For this purpose, Koji and I acquired a couple of old AA boards from the WB electronics shop. These are however, rev2 of the board, whereas the latest version is v6. The main difference between v2 and v6 is that (i) the THS4131 instrumentation amplifier has the Vocm pin grounded in v6 but is floating in v2 and (ii) the buffer opamps are AD8622 in v6 but are AD8672 in v2. But in fact, the boards we have are stuffed with AD8682.
I talked to Rich on Friday, and he seemed to think the AD8672 didn't have any issues noise-wise, the main reason they changed it was because its power consumption was high, and was causing overheating when several of these 1U chassis were packed closely together in an electronics rack. But the AD8682, which is what we have, has comparable power consumption to the AD8622. It is however a JFET opamp, and the voltage noise is a bit higher than the AD8622.
I am sure there is a way to LISO model a differential output opamp like the THS4131, but I thought I'd simulate the noise in LTSPICE instead. But I couldn't get that to work. So instead, I just measured the transfer function and noise of a single channel, for which Koji had expertly hacked together a custom shorting of the THS4131 Vocm pin to ground. Attachments #1 and #2 show the measurement. All looks good. Note that the phase is 180 at DC because I had hooked up the input signal opposite to what it should have been. The voltage noise of the differential outputs (each measured w.r.t. ground, with both inputs shorted to ground by a short patch cable) at 10 Hz is <100nV/rtHz, and the ADC noise is expected to be ~1uV/rtHz, so I think this is fine.
Conclusion: I think for the ALS test, we can just use the AA board in this config without worrying too much about replacing the buffer stage opamps, even though we've ordered 100pcs of AD8622.
Addendum 7 Mar 2018 11am: As per this document, the output noise of the AA board should be <75nV/rtHz from 10 Hz-50 kHz. So maybe the AD8682 noise is a little high after all. I've gotten the LTSpice model working now, will post the comparison of modelled output noise for various combinations here shortly.
In Steve's absence, I've tried to keep an eye on the health of the vacuum system. From Attachment #1, the pressure of the main volume seems stable, no red flags there. I also don't here any anomalously loud sounds near the vacuum pumps. I've changed the N2 cylinders that keep V1 open twice, on Wednesday and Sunday of last week. So in summary, the vacuum system looks fine based on all the metrics I know of.
I spent today making another daughter board (so that we can use the new scheme for I and Q for one arm), testing it (i.e. measuring noise and TF and comparing to LISO model), and arranging all of this inside the 1U demod chassis. To accommodate everything inside, I decided to remove the 2 unused demod units from inside the box. I then drilled a few holes, installed the daughter boards on some standoffs, removed the capacitors and inductors as I outlined yesterday, and routed input and output signals to/from the daughter board. The outputs are routed to a D-sub on the rear panel. More details + better photo + results of testing the combined demod+daughter board signal chain tomorrow...
Since we decided to use the Acromag for readback of the temperature sensor for Kira's seismometer temperature control, I enabled logging of the channel Johannes had reserved for this purpose last week. Kira has made the physical connection of a temperature sensor to the BNC input for this channel - it reads back -2.92 V right now, which is around what I remember it being when Kira was doing her benchtop tests. I edited C0EDCU.ini to enable logging of this channel at 16 Hz. Presumably, a study of the ADC noise of the Acromag at low frequencies has to be made to ensure appropriate whitening (if any) can be added. Channel name is C1:PEM-SEIS_EX_TEMP_MON. Similarly, there is C1:PEM_SEIS_EX_TEMP_CTRL which is meant to be the control channel for the servoing. Calibration of the temperature sensor readback into temperature units remains. It also remains to be verified if we can have these slow EPICS channels integrated with a fast control model, or if the PID temperature control will be purely custom-script based as we have for the FSS slow loop.
I removed the fast channels I had setup temporarily in c1als. Recompilation and restart of the model went smoothly.
we did a bunch of tests to figure out the feasibility of the plan I outlined last night. Bottom line is: we appear to have a working 64 channel ADC (but with differential receiving that means 32 channels). But we need an aLIGO ADC adaptor card (I'm not sure of the DCC number but I think it is D0902006). See attached screenshot where we managed to add an ADC block to the IOP model on c1lsc, and it recognizes the additional ADC. The firmware on the (newly installed) working card is much newer than that on the existing card inside the expansion chassis (see Attachment #1).
Note that we have left the working ADC card inside the c1lsc expansion chassis. Plan is to give Rolf the faulty ADC card and at the same time ask him for a working adapter board.
Unrelated to this work: we have also scavenged 4 pcs of v2 of the differential receiving AA board from WB EE shop, along with a 1U chassis for the same. These are under my desk at the 40m for the moment. We will need to re-stuff these with appropriate OpAmps (and also maybe change some Rs and Cs) to make this board the same as v6, which is the version currently in use.