 Wed Jul 26 19:13:07 2017, Jamie, Update, CDS, daqd showing same instability as before
|
I recompiled daqd on the updated fb1, similar to how I had before, and we're seeing the same instability: process crashes when it tries to write
out the second trend (technically it looks like it crashes while it's trying to write out the full frame while the second trend is also being written
out). Jonathan Hanks and I are actively looking into it and i'll provide further report soon. |
 Thu Jul 27 22:42:24 2017, gautam, Update, SUS, Seismic noise, DAC noise, and Coil Driver electronics noise 
|
Summary:
Yesterday at the meeting, we talked about how the analog de-whitening filters in the
coil driver path may be more aggressive than necessary. I think Attachment |
|
Fri Jul 28 15:36:32 2017, gautam, Update, Optical Levers, Optical lever tuning thoughts
|
Attachment #1 - Measured error signal spectrum
with the Oplev loop disabled, measured at the IN1 input for ITMY. The y-axis calibration into urad/rtHz may not be exact (I don't
know when this was last calibrated). |
|
Fri Jul 28 16:47:16 2017, gautam, Update, General, PSL StripTool flatlined
|
About 3.5 hours ago, all the PSL wall StripTool traces "flatlined", as happens when we had the EPICS freezes in the past - except that all
these traces were flat for more than 3 hours. I checked that the c1psl
slow machine responded to ping, and I could also telnet into it. I tried opening the StripTool on pianosa and all the traces were responsive. |
|
Fri Jul 28 20:22:41 2017, Jamie, Update, CDS, possible stable daqd configuration with separate DC and FW
|
This week Jonathan Hanks and I have been trying to diagnose why the daqd has been unstable in the configuration used by the 40m, with data concentrator
(dc) and frame writer (fw) in the same process (referred to generically as 'fb'). Jonathan has been digging into the core dumps and source
to try to figure out what's going on, but he hasn't come up with anything concrete yet. |
|
Sat Jul 29 14:05:19 2017, gautam, Update, General, PSL StripTool flatlined
|
The PMC was unlocked when I came in ~10mins ago. The wall StripTool traces suggest it has been this way for > 8hours. I was unable to get the PMC
to re-lock by using the PMC MEDM screen. The c1psl
slow machine responded to ping, and I could also telnet into it. But despite burt-restoring |
|
Sat Jul 29 16:24:55 2017, jamie, Update, General, PSL StripTool flatlined
|
Quote:
Unrelated to this work: It looks
like some/all of the FE models were re-started. The x3 gain on the coil outputs of the 2 ITMs and BS, which I had manually engaged when I re-aligned the |
|
Mon Jul 31 15:13:24 2017, gautam, Update, CDS, FB ---> FB1
|
[jamie, gautam]
In order to test the new daqd config that Jamie has been working on, we felt it would be most convenient for the host name "fb" (martian
network IP 192.168.113.202) to point to the physical machine "fb1" (martian network IP 192.168.113.201). |
|
Mon Jul 31 18:44:40 2017, Jamie, Update, CDS, CDS system essentially fully recovered
|
The CDS system is mostly fully recovered at this point. The mx_streams are all flowing from all front ends, and from all models, and the daqd processes
are receiving them and writing the data to frames:
|
|
Mon Jul 31 23:39:02 2017, rana, Update, COC, Cavity Scan Simulation Code
|
Hiro Yamamoto has updated SIS (Static Interferometer Simulation) to allow us to do the MCMC based inference of the 40m arm cavity mirror maps.
The latest version is in git.ligo.org: IFOsim/SIS/
In the examples directory I have put 3 files: |
|
Tue Aug 1 16:05:01 2017, gautam, Update, Optical Levers, Optical lever tuning - cost function construction
|
Summary:
I've been trying
to put together the cost-function that will be used to optimize the Oplev loop shape. Here is what I have
so far. |
|
Tue Aug 1 19:23:06 2017, rana, Update, ALS, X - arm alignment
|
Rana, Naomi
We dither locked the X arm and then aligned the green beam to it using the PZTs. Everything looks ready for us to do a mode scan tomorrow.
We got buildup for Red and Green, but saw no beat in the control room. |
|
Wed Aug 2 09:40:55 2017, Steve, Update, Electronics, spare ILIGO electronics
|
| Spare ILIGO electronics temporarly stored in the east arm. We need cabinet space. |
|
Thu Aug 3 00:59:33 2017, gautam, Update, CDS, NDS2 server restarted, /frames mounted on megatron
|
[Koji, Nikhil, Gautam]
We couldn't get data using python nds2. There seems to have been many problems.
/frames wasn't mounted on megatron, which was the nds2 server. Solution: added |
|
Thu Aug 3 10:51:32 2017, rana, Update, CDS, NDS2 server restarted, /frames mounted on megatron
|
same issue on NODUS; I edited the /etc/fstab and tried mount -a, but it gives this error:
controls@nodus|~ 1> sudo mount -a
mount.nfs: access denied by server while mounting fb1:/frames |
|
Thu Aug 3 11:11:29 2017, gautam, Update, CDS, NDS2 server restarted, /frames mounted on nodus
|
I added nodus' eth0 IP (192.168.113.200) to the list of allowed nfs clients in /etc/exportfs on fb1, and then ran sudo mount -a on nodus. Now /frames
is mounted.
|
|
Thu Aug 3 19:46:27 2017, Jamie, Update, CDS, new daqd restart procedure
|
This is the daqd restart procedure:
$ ssh fb1 sudo systemctl restart daqd_*
That will
restart all of the daqd services (daqd_dc, daqd_fw, daqd_rcv). |
|
Thu Aug 3 20:15:11 2017, Jamie, Update, CDS, dataviewer can not raise test points
|
For some reason dataviewer is not able to raise test points with the new daqd setup, even though dtt can. If you raise a test point with dtt then
dataviewer can show the data fine.
It's unclear to me why this would be the case. It might be that all the versions of dataviewer on the workstations are too old?? |
|
Fri Aug 4 09:07:28 2017, rana, Update, CDS, CDS system essentially NOT fully recovered
|
Tried
getting trends with dataviewer just now since Jamie re-enabled the minute_raw frame writing yesterday. Unable to get trends still:
Connecting to NDS Server fb1 (TCP port 8088) |
|
Fri Aug 4 18:25:15 2017, gautam, Update, General, Bilinear noise coupling
|
[Nikhil, gautam]
We repeated the test that
EricQ detailed here |
|
Sat Aug 5 11:04:07 2017, gautam, Update, SUS, MC1 glitches return
|
See Attachment #1, which is full (2048Hz) data for a 3 minute stretch around when I saw the MC1 glitch. At the time of the glitch, WFS loops were disabled,
so the only actuation on MC1 was via the local damping loops. The oscillations in the MC2 channels are the autolocker turning on the MC2 length tickle.
Nikhil and I tried the usual techniques of squishing cables at the satellite box, and also at 1X4/1X5, but the glitching persists. I will try |
|
Mon Aug 7 16:00:41 2017, rana, Update, General, Bilinear noise coupling
|
These are not the angular test parameters that we're looking for:
recall that what we want is the low frequency beam spot variations and the feedback to be limited to a small high frequency band.
e.g. only inject noise at 40-50 Hz, not loud enough to find at 2x the injected frequency. |
|
Mon Aug 7 22:50:57 2017, Koji, Update, General, New wifi router for the GC network installed
|
I have replaced the old 11n wifi router (CISCO / Linksys) for the GC network with a new one with 11ac technology.
The new one is a 3band wifi router. Thus it has one 2.4GHz (11n) SSID and two 5GHz (11ac) SSIDs. All these have been set to be hidden. Just come
to the 40m and find the necessary info for the connection. |
|
Tue Aug 8 17:04:26 2017, Steve, Update, VAC, unintended pump down
|
IFO pressure 2 Torr, PSL shutter closed. I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses
at atm.
The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later..... |
|
Tue Aug 8 17:44:11 2017, Steve, Update, VAC, unintended pump down
|
Pumpdown stopped for over night at ~ 1 Torr
The roughing line disconnected. Valves condition indicator "moving " means that it is closed and it's cable disconnected
so it can not move. |
|
Tue Aug 8 20:48:06 2017, gautam, Update, SUS, ITMX stuck
|
Somewhere between CDS model restarts and the IFO venting,
ITMX got stuck.
I shook it loose using the usual bias slider technique. It appears to be free now, I was able to
lock the green beam on a TEM00 mode without touching the green input pointing. The ITMX Oplev spot has also returned to within its MEDM display bounds. |
|
Wed Aug 9 11:33:49 2017, gautam, Update, Electronics, MC2 de-whitening
|
Summary:
The analog de-whitening filters for MC2 are different from those on the other optics (i.e. ITMs and ETMs). They have one complex pole pair @7Hz,
Q~sqrt(2), one complex zero pair @50Hz, Q~sqrt(2), one real pole at 2.5kHz, and one real zero @250Hz (with a DC gain of 10dB). |
|
Wed Aug 9 11:59:51 2017, Steve, Update, VAC, unintended pump down at vacuum normal
|
pd80b has reached Vac Normal. IFO pressure 0.5 mTorr
We need our vauum channels back in dataviewer.
|
|
Wed Aug 9 12:05:57 2017, rana, Update, Electronics, data archiving
|
This kind of data fitting and analysis is really useful. We should figure out a way to archive it. Perhaps the data files and fitting stuff can be put
into GIT in some smart way? The fit results can be added to the 40m MC electronics DCC tree. Then the links can be added to this elog. |
|
Wed Aug 9 12:35:47 2017, gautam, Update, ALS, Fiber ALS
|
Last week, we were talking about reviving the Fiber ALS box. Right now, it's not in great shape. Some changes to be made:
Supply power to the PDs (Menlo FPD310) via
a power regulator board. The datasheet says the current consumption per PD is 250 mA. So we need 500mA. We have the D1000217 |
|
Wed Aug 9 15:15:47 2017, gautam, Update, SUS, MC1 glitches return
|
Happened again just now, although the characteristics of the glitch are very different from the previous post, its less abrupt. Only actuation on MC1
at this point was local damping. |
|
Wed Aug 9 16:34:46 2017, rana, Update, VAC, Vacuum Document recovered
|
Steve and I found the previous draft of the 40m Vacuum Document. Someone in 2015 had browsed into the Docs history and then saved the old 2013 version
as the current one.
We restored the version from 2014 which has all of Steve's edits. I have put that version (which is now the working copy) into the DCC: |
|
Wed Aug 9 19:21:18 2017, gautam, Update, ALS, ALS recovery
|
Summary:
Between frequent MC1 excursions, I worked on ALS recovery today. Attachment #1 shows the out-of-loop ALS noise as of
today evening (taken with arms locked to IR) - I have yet to check loop shapes of the ALS servos, looks like there is some tuning to be done. |
|
Thu Aug 10 09:10:55 2017, steve, Update, General, dataviewer is recovering
|
It can look back 7 days trends now. There is still no vacuum channels. I can bring back the channels through the restore directory, but there are no
data. |
|
Thu Aug 10 09:31:57 2017, Steve, Update, SUS, ITMX sensor voltage
|
There must be some bad connection
Quote:
|
|
Thu Aug 10 14:14:17 2017, Kira, Update, PEM, previously built temp sensor
|
I decided to see what was inside the sensor that had been previously made. According to elog 1102, the temperature sensor is LM34, the specs of which
can be found here:
http://www.ti.com/lit/ds/symlink/lm34.pdf |
|
Thu Aug 10 14:25:52 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
I went into /opt/rtcds/caltech/c1/target/daqd,
opened the master
file, and uncommented the line with C0EDCU.ini |
|
Thu Aug 10 15:45:34 2017, Steve, Update, VAC, accidental vent to 17 Torr
|
Finally we got the cold cathode gauge working. IFO pressure 7e-6 Torr-it, vacuum normal valve configuration with all 4 ion pump gate valves closed at
~ 9e-6 Torr. The cryo pump was also pumped yesterday to remove the accumulated outgassing build
up. |
|
Thu Aug 10 21:01:43 2017, gautam, Update, SUS, MC1 glitches debugging
|
I have squished cables in all the places I can think of - but MC1 has been glitching regularly today. Before starting to pull electronics out, I am going
to attempt a more systematic debugging in the hope I can localize the cause.
To this end, I've disabled the MC autolocker, |
|
Fri Aug 11 00:10:03 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Seems like something has failed after I did this - full
frames are no longer on Aug 10 being written since ~2.30pm PDT. I found out when I tried to download some of the free-swinging MC1 data.
To
clarify, I logged into fb1, and ran sudo systemctl restart daqd_*. The only change I made was to uncomment |
|
Fri Aug 11 10:27:49 2017, Kira, Update, PEM, temperature sensor
|
Since there seems to be little difference between AD590 and AD592, I guess we could just go with the AD590. The temperature noise spectrum in the first
graph are for the AD590, so if we want to reproduce those results, we should use AD590.
For the AD581/AD587, we could go with a few varieties that have the least output voltage drift, although I am not sure what precision we will |
|
Fri Aug 11 10:48:39 2017, Kira, Update, PEM, temperature sensor
|
| Quick update: we actually have AD587KRZ and AD592, so we could start by using that and seeing how it works. |
|
Fri Aug 11 11:14:24 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
I commented out the line pertaining to C0EDCU again, now full frames are being written again.
But we no longer have access to the slow EPICS records. |
|
Fri Aug 11 11:56:02 2017, rana, Update, PEM, temperature sensor
|
Might as well order several of a few different varieties today. Its good to have some extra in stock; we don't always want to have to wait days for
parts to show up. If you give Steve a list of parts to buy he can order them today or Monday.
There should also be some precision 5V sources (e.g. AD586) that you can try. |
|
Fri Aug 11 12:27:25 2017, Kira, Update, PEM, temperature sensor
|
Used AD592CNZ and AD586 (5V output) to create a circuit that works and is responsive to temperature changes. At room temp, using ~1K resistor,
it showed ~0.3V across it, as expected. The voltage went up when we heated it with a heating gun. Next step will be to add in an OP amp and design some
experiments to check to see how accurate it is. Thanks to Gautam for helping me with it! |
|
Fri Aug 11 12:32:46 2017, gautam, Update, SUS, MC1 glitches debugging
|
Attachment #1: Free swinging sensor spectra. I havent done any peak fitting but the locations of the resonances seem consistent
with where we expect them to be.
The MC_REFL spot appears to not have shifted significantly (so slow bias voltages are probably not to blame). Now I have to look at trend data |
|
Fri Aug 11 17:36:47 2017, gautam, Update, SUS, MC1 <--> MC3
|
About 30mins ago, I saw another glitch on MC1 - this happened while the Watchdog was shutdown.
In order to further narrow down the cause of the glitch, we switched the Coil Driver Board --> Satellite box DB(15?) connectors on the coil
drivers between MC1 and MC3 coil driver boards. I also changed the static PIT/YAW bias voltages to MC1 and MC3 such that MC-REFL is now approximately back |
|
Fri Aug 11 18:53:35 2017, gautam, Update, CDS, Slow EPICS channels -> Frames re-enabled
|
Quote:
Seems like something has failed after I did this - full frames are no longer on Aug 10 being written since |
|
Fri Aug 11 19:34:49 2017, Jamie, Update, CDS, CDS final bits status update
|
So it appears we now have full frames and second, minute, and minute_raw trends.
We are still not able to raise test points with daqd_rcv (e.g. the NDS1 server), which is why dataviewer and nds2-client can't get test points
on their own. |
|
Sat Aug 12 14:09:36 2017, gautam, Update, SUS, Glitches stay on MC1
|
Even in the switched state, the glitches stayed on MC1.
The coil driver electronics
for MC1, upstream of the Satellite box, was what was previously MC3 electronics.
Attachment #1 shows that there were no glitches in MC3 sensor channels (which are now physically connected to what was previously |