I updated my bandpass filter and have included the bode plot below in Figure 1. It is a fourth order elliptic bandpass filter with a passband ripple of 1dB and a stopband attenuation of 30 dB. It emphasizes the area between 3 and 40 Hz.
Below, I applied this filter to the huddle test data. The results from this were only slightly better in the targeted region than when no pre-filter was applied.
When I pre-filtered the mode cleaner data and then used an IIR wiener filter, I found that the results did not differ much from the data that was not pre-filtered. I'm not sure yet if I'm targeting the right region of this data with my bandpass filter, and will be looking more into choosing a better region. Also, I am only using certain regions of ff when calculating the transfer function, and need to optimize that region also. I uploaded the code I used to make these plots to github.
In the last post concerning the self noise of the accelerometers, I mentioned the differences between the two data sets I was playing with. In order to give a more concrete analysis of the accelerometers self noise, we came to the conclusion that data taking time should be the same.
The plots below show the analysis for the following two datasets:
Old Data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.
Newer data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 57.66 mins worth of data, (we had 20 mins of data in the previous data set).
Filter parameters were kept the same in all calculations, the only change that was added to the analysis was the detrending of the signals using the detrend function on Matlab, this improved the results as the plots show. I also plotted the error bars for the Wiener filtered result for reference as well as the manufactures claimed self noise.
After detrending the data and taking a longer dataset we can see the improvement brought upon by the foam box in the low frequency band of 0.5 - 10 Hz, as shown in the bottom left plot. There is a lot of noise that needs to be cancelled out from 10 Hz and on, which brings to our attention the plot on the bottom right corner. This plot uses the old data but uses all six accelerometers as witnesses, it also improved overall after having detrended the data, but what is peculiar about this plot is the fact that it manages to mitigate the higher frequency 10 - ~100 Hz band noise.
For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?
I've have been talking a little bit with Steve about the seismometer enclosures.
We want to improve on the current stainless steel cans that cover the two Guralps at the end of the arms. In order to do this, we want to cover the interior of the cans with copper foil to improve the thermal conductivity of the enclosure to better control the temperature inside it. Ideally, we would want to copper plate the cans, but cost and difficulty has been an issue.
I have done some rough calculations and it seems that we need a copper layer of thickness being about a third that of the stainless steel can. This happens to be around 0.5-0.6 mm since we have 16 gauge (~1.6 mm) stainless steel cans.
After wrapping the cans interior with copper, we will insulate them with foam in order to improve its thermal inertia. We want to probably use the same foam that Megan has been using for her seismometer enclosure. I have yet to think about a heater, but something similar to Megans resistor thing would work only smaller. I would be placed inside the can, right on the center of its bottom in order to ditribute heat evenly.
I downloaded new accelerometer huddle test data from last night and analyzed it. This new data set is ~55 mins and uses the same Wiener filter parameters as previous huddle test analysis, the main difference being six accelerometers used in the Wiener filter and the improved experimental set up.
After computing the ASD for the self noise for each of the six accelerometers, (being witnessed by the remaining five), we get,
Now computing the mean of the above signals and the corresponding error bars gives the following result,
Comparing to prevoius huddle tests, I can note two trends on the Wiener subtraction:
1) When using six accelerometers, the subtraction above ~8 Hz drastically improves.
2) When using three accelerometers, there is better cancellation in the 1-5 Hz region, see http://nodus.ligo.caltech.edu:8080/40m/11442. This might have to do with how much more closer the accelerometers are to each other?
Specially heavy items: old analoge scope or hardware loaded boxes......etc
The table cover section holding crossbars are not evenly spaced.
You have to center each cover section on the cross bar so it is supported on both sides !
I will clean up on this table tomorrow
Sorry Jamie, I accidentally deleted your elog entry #11453
I was looking at the new seismometer data and plotted the coherence between the different arms of C1:PEM_GUR1 and C1:PEM_GUR2. There was not much coherence in the X arms, Y arms, or Z arms of each seismometer, but there were within the x and y arms of the seismometer.
I think the area we should focus on with filtering is lower ranges, between 0.01 and 0.1, because that it where coherence is most clearly high. It is higher in high frequencies but also incredibly noisy, meaning it probably wouldn't be good to try to filter there.
Koji and Steve,
The result: bad Guralp x-arm cable.
I will swap the short cables tomorrow at the base.
Short 46" long cables at the base plates were swapped. Their solderings looked horrible.
This cable actually worked at 5-5-2015
Bad cable at ETMY station now. The new cable should be a little bit longer ~52"
Koji could pull out easily 11 of the wires from their socket.
I tested both of the front panels (conductive and isolated SMAs) with the ALS Delay Line Box by driving extremely close frequencies through the cables. By doing this, we would expect that a spike would show up in the PSD if there was crosstalk between the cables.
In the plots below, for the conductive panel, the frequencies used were
X Arm: 22.329 MHz Y Arm: 22.3291 MHz
For the isolated panel, the frequencies were
X Arm: 22.294 MHz Y Arm: 22.2943 MHz
This gives a difference of 100 Hz for the conductive panel and 300 Hz for the isolated panel. Focusing on these areas of the PSD, it can be seen that in the Y Arm cable there is a very clear spike within 30 Hz of these differences when frequencies are being driven through both cables as opposed to the signal being in only the Y Arm. In the X Arms, the noise in general is higher when both cables are on, but there is no distinct spike at the expected frequencies. This indicates that some sort of crosstalk is probably happening due to the strong spikes in the Y Arm cables.
Previously, I had gotten the same results for the conductive and the isolated front panels. Today, I sanded off the anodized part on the back of the conductive front panel. I checked afterwards with a mulitmeter to ensure that it was indeed conductive through all the SMA connectors.
I drove a frequency of 29.359 Hz through the X Arm cable and 29.3592 Hz through the Y Arm cable, giving a difference of 200 Hz. Previously, there would only be a spike in the Y Arm at the difference, while the X Arm did not change if the Y arm was on or off. Now that the panel is fully conductive, a spike can also be seen in the X arm, indicating that crosstalk may possibly be happening with this panel, now that the spike corresponds to both the X arm and Y arm. These results are only after one set of data. Tomorrow I'll take two more sets of data with this panel and do a more in depth comparison of these results to what had been previously seen.
Koji had suggested that I sync up the two function generators to ensure that they have the same base frequency and so that crosstalk will actually appear at the expected frequency. After syncing up the two function generators, I drove the following frequencies through each cable:
X: 29.537 MHz Y: 29.5372 MHz
X: 29.545 MHz Y: 29.5452 MHz
Each time, the difference between the frequencies was 200 Hz, so if there was crosstalk, a spike should appear in the PSDs at 200 Hz when frequencies are being driven through both cables simulataneously, but not when just one is on. We very clearly see a spike at 200 Hz in both the X arm and the Y arm with the conductive SMAs, indicating crosstalk. For the front panel with isolated SMAs, we see a spike at 200 Hz when both frequencies are on, but it is much less pronounced than with the conductive SMAs. It seems as though there will be crosstalk using either panel, just less with the isolated SMAs.
Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh.
This continues to happen. I believe the network latency boogeyman is to blame.
There was a long unlocked period because the enable switch for the MC servo fast path (FASTSW) was left off. Running the mcdown script fixed this, but included the error message:
Channel connect timed out: 'C1:IOO-MC_REFL_GAIN' not found.
CA Client Library: Ignored duplicate create channel response from CA server?
which means the IN1 gain didn't get touched. A second pass of the script produced no errors.
I'm thinking of adding some logic that if the autolocker has failed to lock for some period (5 minutes?), it should rerun mcdown.
After testing both the Conductive and Isolated front panels on the ALS delay line box using the actual beatbox and comparing this to the previous setup, I found that the conductive SMAs improved crosstalk the most. Also, as the old cables were 30m and the new ones are 50m, Eric gave me a conversion factor to apply to the new cables to normalize the comparison.
I used an amplitude of 1.41 Vpp and drove the following frequencies through each cable:
X: 30.019 MHz Y: 30.019203 MHz
which gave a difference of 203 Hz.
In the first figure, it can be seen that, for the old setup with the 30m cables, in both cables there is a spike at 203 Hz with an amplitude of above 4 m/s^2/sqrt(Hz). When the 50m cables were measured in the box with the conductive front panel, the amplitude drops at 203 Hz by a factor of around 3. I also compared the isolated front panel with the old setup, and found that the isolated front panel worse by a factor of just over 2 than the old setup. Therefore, I think that using the conductive front panel for the ALS Delay Line box will reduce noise and crosstalk between the cables the most.
I’m working on a code to determine the Gaussianity of a PSD.
It can be difficult to distinguish between GW events and non-Gaussian noise, especially in burst searches. By characterizing noise Gaussianity, we can better recognize noise patterns and distinguish between GW events and noise.
What I did:
I analyzed an hour of S5 L1 data. First, I plotted a timeseries, just to see what I was working with. Then, I produced a PSD (technically, an ASD) for the timeseries using Welch’s method in Python.
I split the data segment into smaller time-chunks and then produced a PSD for each chunk. All PSDs were superimposed in one plot. Here’s a plot for 201 time-chunks of equal length:
For a specific frequency, I can view the spread in PSD value through the production of a histogram.
I’ve made histograms displaying varying PSD values for the 201 PSD plot at 100 Hz, 500 Hz, and 1kHz.
For Gaussian noise, an exponential decay plot is expected. I will continue this analysis by following the statistical method in Ando et al. 2003 to calculate specific values indicative of the Gaussianity of various distributions. I’ll then look at different periods of time in the S5 L1 data to find periods of time suggesting non-Gaussian behavior.
I fixed a shaker that was claimed to be broken. I had to cut the rubber membrane to open the head.
Once it was opened, the cause of the trouble was obvious. The soldering joint could not put up with the motion of the head.
It is interesting to see that the spring has the damping layer between the metal sheets.
After the repair the DC resistance was measured. It was 1.9Ohm. The side of the shaker chassis said "3.5Ohm, Max 15VA". So it can take more than 4A (wow).
I gave 2A DC from the bench top supply and turn the current on and off. I could confirm the head was moving.
I'll claim the use of this shaker for the seismometer development.
I've continued to work on my Gaussianity tests for S5 L1 data.
Following the statistical measure in Ando et al. (2003), I've calculated the Laguerre coefficient, c2, for all frequencies present in my S5 L1 PSD as a metric of Gaussianity. When c2 is zero, the distribution is Gaussian. A positive c2 corresponds to glitchy noise, while a negative c2 suggests stationary noise.
Below is a plot displaying variation in c2 for this PSD:
By observing the c2 value and histogram of distribution of various PSD values at a given frequency, we can elucidate statistical differences in the Gaussian nature of noise at that frequency which are unclear in the standard PSD.
I modified the perl script which does our hourly autoburt so that it can run on megatron instead of op340m (old Solaris machine). Nothing major, just some path stuff. That was the last function of op340m that I know of, so after a week of watching this we ought to be able to power it off and send it to e-waste.
Seems to work so far. It complains about some models that aren't running but mostly it reports successful snapshot taking based on the .req files.
Unfortunately, it seems that its only doing the new target directory, so its missing all of our old VME machines which still use the /cvs/cds/caltech/target area.
But I think Gautam and Jamie and Aidan have volunteered to start our slow controls upgrade by moving the EX slow controls to Acromag and into the new target area. We ought to modify the CRON to point at the old directory for now, but its a temporary fix hopefully.
The MC autolocker and FSSslow scripts weren't running on Megtron. These were started by running the following commands on megatron:
sudo initctl start MCautolocker
sudo initctl start FSSslow
The new autoburt cronjob was failing because the .cron file was not executable (fixed by chmod +x burtnew.cron), and the new perl script didn't use the full path for ifconfig. Similarly, the simulink webview updating script was failing because the full path for matlab wasn't being given. Both of these fixes have been tested and commited to SVN.
chmod +x burtnew.cron
In general, cron scripts can be a real pain, since the cron process doesn't run our .bashrc, and so doesn't know about updates to $PATH, or other environment vairables that get updated through /ligo/cdscfg/workstationrc.sh, which is called by .bashrc. So something that manually works fine in the terminal may not play out as expected when run by cron.
Some things bouncing around my head that haven't made it to ELOG yet:
The new 2x2 fiber couplers arrived today so Eric gave me an overview on the changes to be made to the existing configuration of the FOL fiber box. I removed the box from the table after ensuring that the PDs were powered OFF and removing and capping all fiber leads on the front panel. Here is a summary of the changes made.
I then ran a quick check to see what the power levels were at the input to the PDs, using the fiber coupled power meter. However, I found that there was no light in the fiber marked "PSL light in" (the power meter read out "Sig. Low"). The X arm Aux light had an input power of 1.12 mW, which after the various coupling losses etc went down to 63 uW just before the PD. The corresponding figures for the Y arm are 200 uW and 2.2 uW. I am not too sure of how the AUX light is coupled into fibers so I am not trying to tweak the alignment to see if I get more power.
Eric pointed out that the 1x2 couplers that were used in the previous arrangement and which I recycled, were in fact NOT appropriate - they are not 50-50 couplers but 90-10 couplers, which explains the measured power levels I quoted here.
I switched out these for a pair of the newly arrived 2x2 couplers, and have also replaced the datasheets on the inside of the top cover. I then redid the power level measurements, and got some sensible values this time (see Attachment #1 for revised layout and measured power levels, numbers in red are powers for PSL light, numbers in green are for AUX laser light, and all numbers are in mW). I did find that the 90-10 splitter in the PSL+Y path was not working (though the one in the PSL+X path seems to be working fine), and hence, have not quoted power levels at the output of these splitters. For now, I guess we can bypass the splitters and take the PSL+AUX light from the 2x2 couplers directly to the PDs.
Gautam and Steve,
The decommissioned server from LDAS is retired to the 40m with 32 cores and 128GB of memory in rack 1X7 http://docs.oracle.com/cd/E19121-01/sf.x4600/
We had a look at the IR beat (PSL+Xarm) today using the new FOL fiber box, and compared it to the green beat signal for the same combination. We first switched out the green Y beat input into the RF amplifiers on the PSL table with the PSL+Xarm IR beat input (so in all the plots, the BEATY channels really correspond to the IR beat for PSL+X). The IR and green beat notes were found without much difficulty, and we compared the beat signal PSDs for the green and IR signals (see Attachment #1 - arms were locked to green and the X slow control was turned on). The pink trace (labeled REF1) corresponds to the green beat signal, and was in good agreement with an earlier reference trace Eric had saved for the same signal. The teal trace (labeled REF0) corresponds to the the IR beat signal monitored simultaneously.
We then went back to the PSL table to check the amplitude of the signal from the broadband fiber PDs using the Agilent network analyzer. An initial measurement yielded a beat note (@~50MHz) at ~-22dbm (17mV rms). We figured that by bypassing the 90-10 splitter in this path, we could get a stronger signal. But after switching out the fiber connections we found that the signal amplitude had fallen to ~-27dbm (10mV rms). As per my earlier measurements here, we expect ~600uW of light on the PD, and a quick calculation suggested the signal should be more like 60mV, so we used the fiber power meter to check the power levels after each of the couplers again. We then found that the fiber connector on the front panel of the box for the PSL input wasnt ideal (the laser power after the first 50-50 coupler was only ~250 uW, though the input was ~1.2 mW). The power after the first coupler also fluctuated unpredictably (<100 uW to 350 uW) in response to slightly tightening/loosening the fiber connections on the front panel. I then switched the PSL input to one of the two unused fiber connectors on the front panel (meant for the 10% of the beat signal for the DC readout), and found that this input behaved much better, with ~450 uW of power available after the first 50-50 coupler. The power going into the beat PD was also measured to be ~550uW, closer to what was expected. The beat signal peak now was ~-14dbm (~30mV rms).
We then once again repeated the comparison between green and IR beat signals - but while in the control room, I noticed that the beat signal amplitude on the network analyzer in the control room was fluctuating by nearly 1.5 divisions on the vertical scale - not sure what the reason for this is. A look at the PSD of the IR beat with higher power incident on the PD was also not encouraging (see blue trace in Attachment #1), it seems to have gotten worse in the 10-30 Hz range. We also looked at the coherence between the beat spectrum and the beat note amplitude in order to look for any linear coupling between the two, but from Attachment #2, we cannot explain the disparity between the green and IR beat spectra. This warrants further investigation.
Everything on the PSL table has now been restored to the configurations before these investigations (i.e. the Y+PSL green beat cable has been reconnected to the RF amplifier, and both green beat PDs have been powered back ON. The fiber PDs are powered OFF)
Eric needed a buffer to drive low input impedance (~130Ohm) of his pomona box, I quickly made a non-iverting buffer with G=+10. The DC power is obtained from the back of SR560. It uses 1.02K and 9.09K
to have the gain of ~10. The chip is OP27. In fact this limits the output swing to be +/-5V for the load resistance of 130Ohm. Eric thinks this is enough. If we need more, we need to swap the chip.
As SR560s tend to saturate too quickly, it would be very useful to have this kind of kit in all the labs
once it is packed in a box.
In the last few days, with Koji's help, I have recovered both the FS725 Rubidium references from W. Bridge, one from the ATF lab, and one from the CTN lab. Both are back at the 40m at the moment.
However, the one that was recovered from the ATF lab is no longer locking to the Rubidium reference frequency, although it was locked at the time we disconnected it from the ATF lab. I emailed the support staff at SRS, who seem to think that either the internal oscillator has drifted too far, or the Rb lamp is dead. Either ways, it needs to be repaired. They suggested that I run a check by issuing some serial commands to the unit to determine which of these is actually the problem, but I've been having some trouble setting up the serial link - I will try this again tomorrow. I'm also having trouble generating an RMA number that is needed to start the repair/maintenance process, but I've emailed SRS support again and hope to hear back from them soon.
The other FS725, recovered from the CTN lab earlier today, seems to work fine and is locked to the Rb reference at the moment. I plan to redo the calibration of the phase tracker with an 'absolute' frequency reference with the help of the FS725 and out GPS timing unit tomorrow. Once that is done, the working unit can be returned to the CTN lab.
In order to synchronise the FS725 Rb clock with our GPS timing signals, I laid out a longish cable running from 1X7 to the IOO rack via the overhead cable guide. There was a T-connector attached to the 1pps output of the GPS timing unit, with one of the outputs unused - I have connected one end of the cable I laid out to this output, with the other end going to the 1pps input of the FS725. I am now waiting for the FS725 to sync to the external reference, before running the calibration of the phase tracker once again using the same method detailed here, using the 10MHz output from the FS725 to serve as a reference for the Fluke RF signal generator...
A Caltech maintenance staff dropped by at around noon today, and told me that he had seen a small puddle of water on the other side of the door along the Y-arm that is kept locked (about 10m from the end-table, on the south side of the arm). He suspected a leak in the lab. Koji and I went down to the said door and observed that there was indeed a small puddle of water accumulated there. There isn't any obvious source of a leak on our side of the door, although the walls tiles in the area suggest that there could be a leak in one of the pipes running through the wall/under the floor. In any case, the leak doesn't seem too dramatic, and we have decided to consult Steve as to what is to be done about this once he is back on Wednesday.
The leak was found inside the wall. Fortunately the plumbers were able to access it from CES room 108
This has been leaking for sometimes. The damaged wall area is about 18 ft long and 1 ft high.
A new type of plot is now available for use in the summary pages, based on EricQ's 2D histogram plots (elog 11210). I have added an example of this to the SandBox tab (https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20151119/sandbox/). The usage is straighforwad: the name to be used in config files is histogram2d; the first channel corresponds to the x-axis and the second one to the y-axis; the options accepted are the same as numpy.histogram2d and pyploy.pcolormesh (besides plot limits, titles, etc.). The default colormap is inferno_r and the shading is flat.
Dataviewer x axis end is not there.
On ( 2600 days) longer plots it is missing 8 moths and on (100 days) shorther plot it is missing 1 month.
SLOWDC servo was dead. I followed EricQ's instruction.
MC Autolocker got stack somewhere. I had to go to megatron and kill MC Autolocker.
init relaunched the autolocker automatically, and now it started properly.
[Eric Q, Gautam, Koji]
We went through the network connections to produce the mapping of the instruments.
Gautam summarized the notes into a spread sheet. See attachments.
We didn't find any irregular connections except for the connection of NETMGR port of c1ioo to Martian Network.
This cable was removed.
I have added a new cron job in pcdev1 at CIT using the 40m shared account. This will run the /home/40m/DetectorChar/bin/cleanarchive script one minute past midnight on the first of every month. The script removes GWsumm archive files older than 1 month old.
However, the one that was recovered from the ATF lab is no longer locking to the Rubidium reference frequency, although it was locked at the time we disconnected it from the ATF lab. I emailed the support staff at SRS, who seem to think that either the internal oscillator has drifted too far, or the Rb lamp is dead. Either ways, it needs to be repaired. They suggested that I run a check by issuing some serial commands to the unit to determine which of these is actually the problem, but I've been having some trouble setting up the serial link - I will try this again tomorrow.
The Rubidium standard we had sent in for repair and recalibration has come back. I checked the following:
However, I am still having trouble setting up a serial communications link with the FS725 with a USB-serial adaptor - I've tried with a Raspberry Pi and my Mac (using screen to try and connect), and also using one of the old Windows laptops lying around on which I was able to install the native software supplied by SRS (still using the USB-serial adaptor to establish connection though). Could it be that the unit is incompatible with the USB-serial adaptor? I had specifically indicated in the repair request that this was also a problem. In any case, this doesn't seem to be crucial, though it would have been nice for diagnostics purposes in the future...
I've stored the repaired FS725 inside the electronics cabinet (marked "Eletronics Modules") for now (the other unit was returned to Antonio in W. Bridge some weeks ago).
PMC locked manually and PRM sus damping restored.
Projector light bulb ordered
[ Manasa, Ericq and Steve ]
Vivitek D952HD with 186 hours installed.
The new wifi router, a Netgear R6400, has been installed, next to the old one which is disconnected (but not yet removed).
Same SSID, and I've added only the wireless MAC addresses of viviana, paoloa and asia, the three thinkpads inside.
Qualitatively, dataviewer at the X end seems pretty snappy. I'll do some more quantitative comparison of the two routers at some point soon. I will update the wiki, too.
I configured a new wifi bridge for a GPIB Instruments.
The some facts are described on https://wiki-40m.ligo.caltech.edu/Network
The setting up wasn't so straight forward. I added more details there as a linked page.
One thing I had to do with the martian wifi router was that I had to separate the name of SSIDs for 2GHz and 5GHz networks.
Now the data download from Agilent is super fast!
The first establishing the connection takes the most of the time, and the data transfer takes literary nothing.
controls@pianosa|netgpibdata > time ./netgpibdata -i 192.168.113.167 -d AG4395A -a 10 -f meas01
Connecting to host 192.168.113.167, GPIB 10...
Data will be written into meas01.dat.
Parameters will be written into meas01.par.
Writing measurement data to file...
Writing to the parameter file.
I came into the 40m a few minutes ago, and noticed the following (approximately in this order):
To investigate further, I checked today's summary pages, and whatever caused this, happened around 730am today morning (approx 5 hours ago). I also saw that all the watchdogs were tripped, except MC3, BS and SRM.
I then tracked down the beeping - I believe that it is coming from Megatron.(in fact, it is coming from the Jetstor..)
I also found that the PSL is OFF, and the Marconi, though ON, has the display parameters set to values that I normally see when it is first turned ON (i.e. the carrier frequency is 1200MHz, the output is -140dBm etc - this is what led me to suspect that somehow the power connection was interrupted? As far as the workstation computers are concerned, I don't think ROSSA was affected, but pianosa is frozen and donatella is at the login screen. The CDS overview MEDM screen refuses to load correctly (though some of the other MEDM screens are working fine). I'm not entirely sure how to go about fixing all of this, so for now, I'm leaving the PSL off and I've shutdown the remaining watchdogs.
It just occurred to me to check the status of the vacuum - the MEDM screen seems to suggest everything is fine (see Attachment #1). I went down to the X end to do a quick check on the status of the turbo pumps and everything looks normal there...
Looks like that's the case. LIGO GC also sent an e-mail that there was a popwer glitch.
Pasadena reported the Sunday night event as a power transient caused by the trip of a power transmission line. This affected the entire city. Once the loss was detected, the system automatically switches to an alternate source. It was about one second for the system to recover.
2W Innolight, both Lightwaves at the ends, PSL HEPA, Ref Cavity and 3 AC units turned on.
The 40m vacuum did not trip. It is vacuum normal.
The Jetstore computer is indicating power failer.
Chiara reports an uptime of >195 days, so its UPS is working fine
FB, megatron, optimus booted via front panel button.
Jetstor RAID array (where the frames live) was beeping, since its UPS failed as well. The beep was silenced by clicking on "View Events/Mute Beeper" at 192.168.113.119 in a browser on a martian computer. I've started a data consistency check via the web interface, as well. According to the log, this was last done in July 2015, and took ~19 hrs.
Frontends powered up; models don't start automatically at boot anymore, so I ran rtcds start all on each of them.
rtcds start all
All frontends except c1ioo had a very wrong datetime, so I ran sudo ntpdate -b -s -u pool.ntp.org on all of them, and restarted the models (just updating the time isn't enough). There is an /etc/ntp.conf in the frontend filesystem that points to nodus, which is set up as an NTP server, but I guess this isn't working.
sudo ntpdate -b -s -u pool.ntp.org
PMC locking was hindered by sticky sliders. I burtrestored the c1psl.snap from Friday, and the PMC locked up fine. (One may be fooled by the unchanged HV mon when moving the offset slider into thinking the HV KEPCO power supplies need to be brought down and up again, but it's just the sliders)
Mode cleaner manually locked and somewhat aligned. Based on my memory of PMC camera/transmission, the pointing changed; the WFS need a round of MC alignment and WFS offset setting, but the current state is fine for operation without all that.
I was able to realign the arms, lock them, and have run the dither align to maximize IR transmission - looks like things are back to normal now. For the Y-end, I used the green beam initially to do some coarse alignment of the ITM and ETM, till I was able to see IR flashes in the control room monitors. I then tweaked the alignment of the tip-tilts till I saw TEM00 flashes, and then enabled LSC. Once the arm was locked, I ran the dither align. I then tweaked ITMX alignment till I saw IR flashes in the X arm as well, and was able to lock it with minimal tweaking of ETMX. The LSC actuation was set to ETMX when the models were restarted - I changed this to ITMX actuation, and now both arms are locked with nominal IR transmissions. I will center all the Oplev spots tomorrow before I start work on getting the X green back - I've left the ETM Oplev servos on for now.
While I was working, I noticed that frame builder was periodically crashing. I had to run mxstream restart a few times in order to get CDS back to the nominal state. I wonder if this is a persistent effect of the date/time issues we were seeing earlier today?
daqd has indeed continued to be unstable. I found system times had drifted apart again... I think something weird happened in the booting of the frontends. The monit processes were not running on any of the frontends. I ntpdate'd again, and manually started monit on each fronted via sudo /etc/init.d/monit start.
sudo /etc/init.d/monit start
10:15 power glitch today. ETMX Lightwave and air conditions turned back on
The CDS situation was not as catastrophic as the last time, it was sufficient for me to ssh into all the frontends and restart all the models. I also checked that monit was running on all the FEs and that there was no date/time issues like we saw last week. Everything looks to be back to normal now, except that the ntpd process being monitored on c1iscex says "execution failed". I tried restarting the process a couple of times, but each time it returns the same status after a few minutes.