I wired all 32 channels going to the AA board directly to the ADC as described in the previous log. However, instead of using the old AA board and bypassing the whole circuit, I just used a breakout board as is shown in the first attachment. I put the board back in the rack and reconnected all of the cables.
The seismic BLRMs appear to be working again. A PSD of the BS seismometers is shown in attachment 2. Tomorrow I'll look at how much the ADC alone is suppressing the common mode 60 Hz noise on each of the channels.
Steve: 5 of ADC DAC In Line Test Boards [ D060124 ] ordered. They should be here within 10 days.
Increased the Integral gain (from -1 to -4) on the EX temperature controller. This didn't work a few weeks ago, but now with the added P gain, it seems stable. Daily temperature swings are now ~3x smaller.
Notes for Kira on what we need to do tomorrow (Friday):
For those who are flabbergasted by the way I calibrated the TEMP_MON channel from volts to deg C, here's how:
use the 'scale' and 'translate' fields to change the slope and offset for calibration in the obvious ways
Yesterday I wired the outputs from the seismometers directly to the ADC input bypassing the old AA board circuit as is described in this elog. The old circuit converted the single-ended output from the seismometers to a differential signal. Today I looked at whether 60 Hz noise is worse going directly into the ADC due to the loss of the common mode rejection previously provided by the conversion to differential signals.
I split the output from the BS Z seismometer to the new board and to an SR785. On the SR785 I measured the difference between the inner and outer conductors of the seismometer output, i.e. A-B with A the center conductor and B the outer conductor, with grounded input. At the same time I took a DTT spectrum of C1:PEM-SEIS_BS_Z_IN1. Both spectra were taken with 1 Hz bandwidth and 25 averages. The setup is shown in attachment 1.
The spectra are shown in attachment 2. The DTT spectrum was converted from counts to volts by multiplying by 2 * 10 V/32768 cts where the extra factor of 2 is from converting from single-ended to differential input. If there was common 60 Hz noise that the ADC was picking up we would expect to see less noise at 60 Hz in the SR785 spectrum measured directly at the output from the seismometer since that was a differential measurement. Since both spectra have the same 60 Hz noise, this noise is differential.
After fixing the spelling of the EX temperature readback, I also added all of the MEDM sliders to the C0EDCU.ini file (making sure to add an even number of channels). Restarted FB (after installing telnet on rossa):
telnet fb 8083
preferred method of posting DataViewer images: print as a SVG image (since its vectorized). Then from the command line do:
inkscape steven.svg --export-pdf=vass.pdf
for whatever reason, I am unable to get minute or second trends from nodus for any channels (IMC, PEM, etc) since the reboot. has there been some more recent FB failure or is this still a bug since last years FB catastrophe?
I've attached a sketch of how the panel will be mounted. We should make a small rectangular box that would raise the panel from the block by 1 cm or so to allow the cables to fit into the hole in the block without getting bent. It also has to be airtight so maybe having a thin layer of rubber between the mount and block would be good.
I added an out of loop sensor to the can by placing the lab temperature sensor inside the can. I'm not sure which channel is logging this temperature though. I also noticed that the StripTool still had the old misspelled name for the temperature readout so I fixed that as well.
I've attached a picture of the setup.
I connected up the channels for the ADC Acromag a while back and we were planning to install it today so that we could set up a new channel for the out of loop sensor. Unfortunately, the Acromag seems to be broken. We connected up a precision 10V voltage to one of the channels, but the Acromag read out ~7V and it kept fluctuating. Even after calibration, we still got the same result. When enabling the legacy support, we got ~11V. But when we measured the voltage at the screw terminals with a multimeter and it showed 10V, so the issue is not with the wiring. All of the channels have this same issue. We will be ordering more Acromags soon, so hopefully we'll be able to set up the channel soon. I've attached a picture of the Acromag along with the front panel with the channels labeled
As part of investigation into this issue, Jonathan Hanks pointed out that the "minute trends" being recorded by our system were actually only being recorded every 120 seconds (a.k.a. 2 minutes). He had fixed the appropriate line in the parameter file, but had not restarted the FW processes. I had restarted it on Friday. (but failed to elog it !)
To check if this made any difference, I pulled 1 hour of "minute trend" data for the PSL table temperature channel from ~1 hour ago, and compared the number of datapoints against a 1 hour minute trend time series from 2 May. I've put the display with # of datapoints (for an identical length of time) from before [left] and after [right] the restart next to the plots in Attachment #1. Seems like we are getting minute trends written every 60 seconds now, as it should be .
The unavailability of trends from nodus is a separate issue for which JH has suggested another fix, to be elogged separately.
Here are a few things I will be working on:
The replacement Acromag we scooped from the West Bridge E-Shop does actually seem to work, although we thought it was broken - at first it was just outputting zeros, but after I did the calibration procedure, applying +10 V and -10 V, respectively, it was reporting voltage correctly, over the full range. I don't know why the factory settings would be messed up, but it had been out of the box before. I did this only with channel 7, so you need to calibrate channels 0-6 and confirm that they indeed also work properly.
I tried calibrating the other channels today, but they still fluctuate. Sometimes they do stabilize at +/- 10V, but then suddenly drop to 5 or 6 V before climbing back up to 10. Turning the legacy off made it go only up to 6.67V. This happens for all the channels, even after doing a factory reset and recalibrating. Not sure what's happening here.
As described in this elog, the ADC for the seismometers now has the signals wired directly to the ADC instead of going through an AA board or other circuit to remove any common mode noise. This elog describes one test of the common mode rejection of this setup. Guantanamo suggested comparing directly with a recent spectrum taken a few months before the new setup described in this elog.
Today I took a spectrum (attachment 1) of C1:PEM-MIC_2 (Ch17) and C1:PEM-MIC_3 (Ch18) with input to the ADC terminated with 50 Ohms. These are two of the channels plotted in the previous spectrum, though I don't know how that plot was normalized. It's clear that there are now strong 60 Hz harmonic peaks that were not there before, so this new setup does have worse common mode rejection.
Earlier today Kira and I reconnected the EX seismometer. I just took some spectra of all three seismometers, shown in the attachments, to compare with past data and to do a rough check of the calibration.
This elog has a spectra from 2010 (GUR1 is now EY) and this elog has one for BS at lower frequencies from 2017. Note that the EX seismometers now have strong peaks that are not at 60 Hz harmonics. Other than these peaks, these old spectra roughly match up with the ones taken today, so the callibration is still roughly the same. I couldn't find any old data for EX (GUR2) though so I don't know for sure that these peaks weren't there before.
gautam 20180517 0930: In 2017, Gur2 (now EX) looked like this. Still peaky, but the peaks seem shifted in frequency. Steve also informed me that the Gur1 and Gur2 cables were swapped n times, so perhaps we shouldn't read too much into that.
I've moved my setup to the actual seismometer. I attached the temperature sensor to the seismometer (attachment 1) with duct tape, though this is temporary. I will be monitoring the temperature fluctuations of the seismometer for a whole day then take the can off and repeat the test. The can isn't clamped down so the insulation isn't perfect, so I'd expect to see some noticeable fluctuations even with the can on. I've also labeled the long cable for the temperatuse sensor readout (attachments 2 and 3). There will also be an out of loop sensor added in later, but for this test since I am not running the loop it doesn't matter which sensor I monitor. Attachment 4 is a picture of the current setup.
Here is the result of my test. I think I'll leave the can on over the weekend because there's a long period of time where the seismometer heated up by 0.8 degrees so I can't fully see the fluctuations over a full 24 hour period.
I guess it's fine for now while we are still finalizing the setup at EX, but we should eventually line up the seismometer axes with the IFO axes. Is there a photo of the orientation of the seismometer pre heater can tests? If not, probably good to make some sort of markings on the granite slab / seismometer to allow easy lining up of these axes...
I have attached the graph for the seismometer temperature fluctuations over 3 days. As we can see, there is a noticeable fluctuation in daily temperature as well as a difference between days in the maximum and minimum temperatures. I will repeat this test but take the can off to see if there's any difference between having the can on or off.
It appears that one of the wires was disconnected overnight or this morning so I wasn't able to gather data over a full 24 hour period. Perhaps someone accidentally kicked it. I placed some cones in that area so hopefully the wires won't be in the way as much and I can get the data tomorrow. From the data I do have it seems that the seismometer is at a colder temperature when the can is not on, though it is difficult to see by how many degrees the temperature fluctuates. I've included the data from 5 days back to see the comparison.
This time the test went without issue. The first attachment is the data for the past 24 hours and the second attachment is the full data over 6 days. The average temperature fluctuations (from highest point to lowest point) for the can on was 0.43 C and for the can off it came out to 0.55 C. In addition the seismometer with the can off is about 1 C cooler than with the can on. I'd like to leave the can off until the end of the week so we can get a comparable data set for both the can on and off. Eventually I'll need to figure out a way to clamp the can down to the block in order to get better insulation and hopefully get even smaller temperature fluctuations.
Rana said that it wasn't necessary to gather more data on the temperature fluctuations so I have reconnected the heater circuit and restarted the PID loop with the can on the seismometer.
We will need to order a few things for our final setup.
I've updated the parts list to be an excel document and included every single part we will need. This is ony a first draft so it will probably be updated in the future. I also made a mistake in hole sizing for the front panel so I've updated it and attached it as well (second attachment).
Edit: re-attached the EX can panel fpd file so that everything is in one place
Chris replaced some air condition filters and ordered some replacement filter today.
Yesterday morning was dusty. I wonder why?
The PRM sus damping was restored this morning.
Yesterday afternoon at 4 the dust count peaked 70,000 counts
Manasa's alergy was bad at the X-end yesterday. What is going on?
There was no wind and CES neighbors did not do anything.
Air cond filters checked by Chris. The 400 days plot show 3 bad peaks at 1-20, 2-5 & 2-19
We noticed quite a strong burning smell in the office area and control room ~20mins ago. We did a round of the bake lab, 40m VEA and the perimeter of the CES building, and saw nothing burning. But the smell persists inside the office area/control room (although it may be getting less noticeable). There is a whining noise coming from the fan belt on top of the office area. Anyways, since nothing seems to be burning down, we are not investigating further.
Steve [ 10am 5-31 ] we should always check partical count in IFO room
I have attached the result of running the PID script on the seismometer with the can on. The daily fluctuations are no more than 0.07 degrees off from the setpoint of 39 degrees. Not really sure what happened in the past day to cause the strange behavior. It seems to have returned back to normal today.
Got this 1U box from the Y arm that we could potentially use (attachment 1). It doesn't have handles on the front but I guess we could attach them if necessary. Attachment 2 is a switch that could be used instead of a light up switch, but now we need to add LEDs on the front panel that indicate that the switch is functional. Attachment 3 is a terminal block that we can use to attach the 16 gage wire to since it is thick and attaching it directly to the board would be difficult. If this is alright to use then I'll change up my designs for the front panel and PCB to accomodate these parts.
We have 6 of these boards now in cabinet E7
Since I am finishing my job at the lab, I have stored all my electronics in a box (attachment 1) and placed it under the table in the control room where some other electronics are stored. The box contains the heater circuit box, two temperature sensor boards, one temperature sensor, a short power cable and +/- 15V supply cables. In the lab I left the wires for the current setup and tied them down to the wall so that they aren't in the way (attachment 2). I left the can as is and the other temperature sensor is still attached to the inside of the can. I have labeled the wires going from the sensor as 'in' and 'out'. I've also left the wires for the heater there as well (attachment 3). I turned off the PID control and deactivated the tmux session on megatron.
Thanks to Rana and the LIGO team for giving me the opportunity to work at the 40m on this project with the seismometer.
You should wipe off the table cover before you take it off next time.
It is important to turn up the PSL encloure HEPA Variac voltage if you are working in there. It takes less than 10 minutes to reach lab condition.
Lab air count normal. It is not logged. I have a notebook of particle count on the SP table next to the Met One counter.
We (Rana and I) are re-assembling the temperature controls on the seismometer to attempt PID control and then improve it using reinforcement learning.
We tried to re-assemble the connections for the heater and in-loop temperature sensor on the can that covers the seismometer.
We fixed (soldered) two of the connections from the heater circuit to the heater, but did not manage to get the PID working as one of the wires attached to the MOSFET had come off. Re-soldering the wire would be attempted tomorrow.
Equipment for undertaking all this is still left at the X-end of the interferometer and will be cleared soon.
After removing all the clamping screws from the heater circuit board, I soldered the wire connecting IRF630 to the output of OP27, which had come off earlier. This can only be a temporary fix as the wire was not long enough to be able to make a proper solder joint. I also tried fixing two other connections which were also almost breaking.
After re-assembling everything I found out that one of the LEDs was not working. The most likely cause seems to be an issue with LM791, LM 781 or the LED itself. Due to the positioning of the wires, I was unable to test them today but will try again possibly tomorrow.
Equipment used for this is still lying at the X end.
Earlier today I cleared up most of the equipment at the X end near the seismometer to make the area walkable.
In the process, I removed the connections to the temperature sensor and placed the wires on top of the can.
While Shruti is re-building Kira's heater circuit, I looked up how to do one of these (i.e. what does a real EE say about how to build a current source?):
It turns out that there is an Analog Devices application note (AN-968) about this (as there usually is once we get tired of playing around and try to look up the right answer).
I've linked to the note and attached the recommended schematic for high current applications. We'll go ahead as is, but we'll make a PCB according to this App Note for the v3 circuit.
Small earth quakes and suspensions. Which one is the most free and most sensitive: ITMX
The rat is cut by mechanical trap and it was removed from ITMX south west location.
A nagy kover patkanyt a fogo elkapta es megolte.
Physical plan is cleaning our roof and gutters today.
Exceptions: cryo pump and 4 ion pumps
Vac Status: The vac rack power was recycled yesterday and power to controller TP1,2 and 3 restored. atm3
VME is OFF. Power to all other instrument are ON. 23.9Vdc 0.2A
ETMY sus tower with locked optic in HEPA tent at east end is standing by for action.
Yesterday, Koji and I noticed (from the wall StripTool traces) that the vertex seismometer RMS between 0.1-0.3 Hz in the X-direction increased abruptly around 6pm PDT. This morning, when I came in, I noticed that the level had settled back to the normal level. Trending the BLRMS channels over the last 24 hours, I see that the 0.3-1 Hz band in the Z direction shows some anomalous behaviour almost in the exact same time-band. Hard to believe that any physical noise was so well aligned to the seismometer axes, I'm inclined to think this is indicative of some electronics issues with the Trillium interface unit, which has been known to be flaky in the past.
I looked into the seismometer situation a bit more today. Here is the story so far - I think more investigation is required:
Attachment #2 has some spectrograms (they are rather large files). They suggest that the increase in noise in the 0.1-0.3 Hz band in the BS seismometer X channel is real - but there isn't a corresponding increase in the other two seismometers, so the problem could still be electronics related.
The Trillium T240 seismometer needs mass re-centering. Has anyone done this before, and do we have any hardware to do this?
I went to the Trillium interface box in 1X5. In this elog, Koji says it is D1000749-v2. But looking at the connector footprint on the back panel, it is more consistent with the v1 layout. Anyway I didn't open it to check. Main point is that none of the backplane data I/O ports are used. We are digitizing (using the fast CDS system) the front panel BNC outputs for the three axes. So of the various connectors available on the interface box, we are only using the front panel DB25, the front panel BNCs, and the rear panel power.
The cable connecting this interface box to the actual seismometer is a custom one I believe. It has a 19 pin military circular type hermetic connector on one end, and a DB25 on the other. Power is supplied to the seismometer from the interface box via this cable, so in order to run the test, I had to use a DB25 breakout board to act as a feedthrough and peek at the signals while the seismometer and interface boards were connected. I used Jenne's mapping of the DB25--> 19 pin connector (which also seems consistent with the schematic). Findings:
I am holding off on attempting any re-centering, for more experienced people to comment.
I removed the Trillium T240 DAQ interface unit from 1X4 for investigation.
It was returned to the electronics rack and all the connections were re-made. Some details:
Update 445pm: Seems to have done something good - the old feedforward filters reduce the YAW RMS motion by a factor of a few. Pitch performance is not so good, maybe the filter needs re-training, but I see coherence, see Attachment #2 for the frequency domain WF.
Attachment #1 shows the spectra of our three available seismometers over a period of ~10ksec.
Attachment #2 shows the result of applying frequency domain Wiener filter subtraction to the POP QPD (target) with the vertex seismometer signals as witness channels.
this is due to the Equivalence Principle: local accelerations are indistinguishable from spacetime curvature. On a spherical Earth, the local gradient of the metric points in the direction towards the center of the Earth, which is colloquially known as "down".
I don't understand why the z-axis motion reported by the T240 is ~10x lower at 10 mHz compared to the X and Y motions. Is this some electronics noise artefact?
Here is some disturbance in the spacetime curvature, where the local gradient of the metric seems to have been modulated (in the "downward" as well as in the other two orthogonal Cartesian directions) at ~1 Hz - seems real as far as I can tell, all the suspensions were being shaken about and all the seismometers witnessed it, though the peak is pretty narrow. A broader, less prominent peak also shows up around 0.5 Hz. We couldn't identify any clear source (no LN2 fill-up / obvious CES activity). This event lasted for ~45 mins, and stopped around 2315 local time. Shortly (~5min) after the ~1 Hz peak died down, however, the 3-10 Hz BLRMS channel reports an increase by ~factor of 2.
Onto trying some locking now that the suspensions have settled down somewhat.
at 1 Hz' this effect is not large so that's real translation. at lower frequencies a ground tilt couples to the horizontal sensors at first order and so the apparent signal is amplified by the double integral. drawing a free body diagram u can c that
x_apparent = (g / s^2) * theta
but for vortical this not tru because it already measures the full free fall and the tilt only shows up at 2nd order
The large ground motion at 1 Hz started up again tonight at around 23:30. I walked around the lab and nearby buildings with a flashlight and couldn't find anything whumping. The noise is very sinusoidal and seems like it must be a 1 Hz motor rather than any natural disturbance or traffic, etc. Suspect that it is a pump in the nearby CES building which is waking up and running to fill up some liquid level. Will check out in the morning.
Estimate of displacement noise based on the observed MC_F channel showing a 25 MHz peak-peak excursion for the laser:
dL = 25e6 * (13 m / (c / lambda)
= 1 micron
So this is a lot. Probably our pendulum is amplifying the ground motion by 10x, so I suspect a ground noise of ~0.1 micron peak-peak.
(this is a native PDF export using qtgrace rather than XMgrace. uninstall xmgrace and symlink to qtgrace.)
Attachment #1 is a spectrogram of the BS sesimometer signals for a ~24 hour period (from Wednesday night to Thursday night local time, zipped because its a large file). I've marked the nearly pure tones that show up for some time and then turn off. We need to get to the bottom of this and ideally stop it from happening at night because it is eating ~1 hour of lockable time.
We considered if we could look at the phasing between the vertex and end seismometers to localize the source of the disturbance.