40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 77 of 339  Not logged in ELOG logo
ID Date Author Type Category Subject
  13213   Wed Aug 16 14:57:01 2017 SteveUpdatePSLref Cavity heating blanket power supply

The last entry I found relating to ref cavity was 2011 Aug 19

Attachment 1: refCavPSheat.jpg
refCavPSheat.jpg
Attachment 2: refCavTempSens.jpg
refCavTempSens.jpg
Attachment 3: refTempSen_1X1.jpg
refTempSen_1X1.jpg
  13212   Wed Aug 16 14:54:13 2017 gautamUpdateCDSPSL monitoring Acromag EPICS server restarted

[johannes, gautam, jamie]

  • Made a directory /opt/rtcds/caltech/c1/scripts/Acromag/PSL where I copied over the files needed my modbusApp to start the server from Lydia's user directory
  • Edited /ligo/apps/ubuntu12/ligoapps-user-env.sh to export a couple of EPICS variables to facilitate easy startup of the EPICS server
  • Started a tmux session on (soon to be re-christened?) megatron called "acroEPICS"
  • Ran the following command to start up the EPICS server:
${EPICS_MODULES}/modbus/bin/${EPICS_HOST_ARCH}/modbusApp npro_config.cmd

To do:

  1. Make a startup script that runs the above command - eventually this can contain the initialization instructions for all the Acromags
  2. Figure out the initctl/systemctl stuff to make the server automatically restart if it drops for some reason (e.g. power failure)
  13211   Tue Aug 15 16:32:42 2017 Jamie, GautumUpdateCDSGPS receiver apparently set to correct mode as "UTC"
Quote:

The setting when we found it was "GPS", which seems logical enough.  However, when we switched it to "UTC" the time as shown on the front panel was correct, now with "UTC" vertically to the right of the time, and fb1 was then showing the correct GPS time.

From Keith Thorne:

In the GPS receiver, you are trying to match the IRIG-B output format that is created by the aLIGO IRIG-B Fanout.  Since we have to prep the aLIGO IRIG-B Fanout every time there is a leap second coming, I would suspect that we are sending UTC to the IRIG-B receivers.  Thus, the GPS receiver needs to be set to that mode.

Soooo, "UTC" is the correct mode for the GPS receiver.

  13210   Tue Aug 15 13:32:38 2017 KiraUpdatePEMtemperature sensor

Tested to make sure that even when only the AD586 was heated that there was no change in the reading. I did so by placing the AD586 away from the rest of the circuit and blowing hot air only on it. There was, in fact, no change. 

Quote:

I didn't realize that the LT1012 needed an additional input to function. I added in +15V and -15V to pins 7 and 4, respectively and placed a 10k resistor and the numbers make more sense now. The voltage showed a negative value, but it became more negative as I heated it up (it's negative due to how a transimpedance amplifier works).

I have attached the new setup and the value it shows (~-3V). It became more negative by about 0.4V, which translates to about a 40K increase in temperature, which makes sense.

In addition, I have attached an updated sketch of the circuit. I will need to do more testing to determine how accurate this is. The next step would be to calculate how much noise there is currently and figure out how to remove this circuit from the breadboard and use a PCB or something like that for final testing in an insulated container.

The reason I chose AD743 initially for the OP amp is because at low frequencies (which is what we are working with), a FET amp such as AD743 will have a low current noise at high impedance, which is what we have in this case. While a FET amp has high voltage noise compared to other OP amps, the current noise becomes more important at high impedance, so it will work better. According to Zach's graphs, the AD743 is best at high impedances, followed by LT1012.

Quote:

Decided to try adding in an OP amp just to see if it would work. Added LT1012 and a 100k resistor to the circuit (I originally wanted to do AD743 as it seems to be the best choice according to Zach's elog here, but it said that they are very precious so I went with LT1012 for testing purposes). When heating it with a heating gun, the output voltage went down by a few 0.01V. The maximum voltage was 0.686V. Similar thing happened when I switched to a 10k resistor, where the maximum was 0.705V and it also went down by a few 0.01V upon heating.

I've attached a few pictures showing the circuit.

 

 

  13209   Tue Aug 15 11:50:21 2017 KiraSummaryPEMtemp sensor packaging/mount

For the final packaging/mounting of the sensor to the seismometer, I have thought of two options.

1. Attach circuit to a PCB board and place it inside the can, while leaving the AD590 open to the air inside the can.

  • This makes sure that the sensor gets a direct measurement of the temperature of the air in the can, as it is exposed to the air. 
  • But, it takes a limited area of measurement, so it could be the case that the area we place it in happens to be a hot or cold pocket, and the measurement would be inaccurate.
  • This can be solved by placing multiple copies of the circuit in various places of the can and averaging the values.

2. Attach the AD590 to a copper plate with thermal paste and put it into a pomona box.

  • This solves the problem of having a limited sample area the first option had, as the copper plate should have a uniform temp distribution, thus we are sampling the temp of that whole area.
  • Need to make sure that the response time to the temperature variations of copper is less than the frequency that we are measuring.
  • This can be calculated using equations for heat transfer (listed below).

If anyone has input on which method is preferred or any additional options that we may have, I would appreciate it.

Heat transfer:

q = k A dT / s

  • k = thermal conductivity
  • A = area
  • dT = temperature gradient
  • s = thickness

For copper, k = 401 W/mK, x = 1.27 mm, A = 2.66x10^-3 m^2 (for the particular copper plate I measured), dT = 1K (assume). Thus the heat transfer will be 839 J/s.

I'm not completely sure what to do with this yet, but it could help us decide whether the copper plate option will be useful for us.

 

  13208   Tue Aug 15 08:22:16 2017 SteveUpdateGeneralprojector light bulb replaced

BL-FS300C-PH-LE was replaced after 2,904 hrs  It did not explode this time.  The 4 monts life period is actually pritty good because this is a $73.  cheap bulb. The best-high priced warranty is 5 months.

 

PS: future option_bulbless laser projector

HITACHI LP-WU3500 PROJECTOR	$2,549.00	
DLP, WUXGA, 3500 LUMENS, HDMI, 20K HR LASER, 5YR WARRANTY, 1.36-2.34:1 LENS, 24/7 DUTY CYCLE
45x72 IMAGE FROM 8' TO 13'10" LENS TO SCREEN, AND AT 10' APPROX. 154FL OF BRIGHTNESS ON A 1.0 GAIN SCREEN
http://www.projectorcentral.com/pdf/projector_spec_9722.pdf

This would give you everything you are requesting, plus a lamp-less design and 5yr warranty.  Ground shipping would be free anywhere in the lower 48, and we would not charge sales tax on orders billing/shipping outside of AZ.  If you have any questions or if you would like to order... just let me know!
  13207   Mon Aug 14 20:12:09 2017 Jamie, GautumUpdateCDSWeird problem with GPS receiver

Today we saw a weird issue with the GPS receiver (EndRun Technologies Tempus LX).  GPS timing on fb1 (which is handled via IRIG-B connection to the receiver with a spectracom card) was off by +18 seconds.  We tried resetting the GPS receiver and it still came up with +18 second offset.  To be clear, the GPS receiver unit itself was showing a time on it's front panel that looked close enough to 24-hour UTC, but was off by +18s.  The time also said "GPS" vertically to the right of the time.

We started exploring the settings on the GPS receiver and found this menu item:

Clock -> "Time Mode" -> "UTC"/"GPS"/"Local"

The setting when we found it was "GPS", which seems logical enough.  However, when we switched it to "UTC" the time as shown on the front panel was correct, now with "UTC" vertically to the right of the time, and fb1 was then showing the correct GPS time.

From the manual:

Time Mode
Time mode defines the time format used for the front-panel time display and, if installed, the optional
time code or Serial Time output. The time mode does not affect the NTP output, which is always
UTC. Possible values for the time mode are GPS, UTC, and local time. GPS time is derived from
the GPS satellite system. UTC is GPS time minus the current leap second correction. Local time is
UTC plus local offset and Daylight Savings Time. The local offset and daylight savings time displays
are described below.

The fact that moving to "UTC" fixed the problem, even though that is supposed to remove the leap second correction, might indicate that there's another bug in the symmetricom driver...

  13206   Mon Aug 14 20:01:38 2017 gautamUpdateSUSGlitches stay on MC1

I don't think we can say for sure. I was just talking to EricQ about this, he said the glitches were often seen when changing the alignment offsets when aligning the arm. I am pretty sure I have seen the ETMX alignment change abruptly since the Ruby Standoff replacement (the Oplev spot just slides across the MEDM display rapidly), but I can't find an elog where I've put in details. I also haven't done a whole lot of work with the arm cavities where I would have noticed this problem. There is this test that Eric did, and it didn't throw up any red flags. But the suspension can be well behaved for weeks at a time before this problem pops up again.

There was also the flaky power connection to the timing card on the ETMX expansion chassis which was fixed only recently, after which there has been no systematic investigation of the status of ETMX.

If it is true that these events are caused by strain building up in the suspension wire, I wonder how we can take systematic steps to avoid it. From what I remember of the SOS assembly procedure, the (unglued) standoff is slid along the optic with the wire under slight tension until the wire slips into the groove on the standoff. Then the tension in the wire is adjusted till the optic is pitch balanced and at the desired height. But it is easy to imagine imprinting some torsional stresses in the (40 um?) wire during this process of looping it around under the optic and placing it in the groove. But perhaps this mechanism makes a negligible contribution to the effect we are seeing, and some other mechanism is responsible in this case.

Quote:

We used to have similar suspension excursion at ETMX. This was the motivation to replace the stand-offs from Al ones to ruby ones. Did the replacement solve the issue at ETMX?

 

  13205   Mon Aug 14 19:41:46 2017 JamieUpdateCDSfront-end/DAQ network down for kernel upgrade, and timing errors

I'm upgrading the linux kernel for all the front ends to one that is supposedly more stable and won't freeze when we unload RTS models (linux-image-3.2.88-csp).  Since it's a different kernel version it requires rebuilds of all kernel-related support stuff (mbuf, symmetricom, mx, open-mx, dolphin) and all the front end models.  All the support stuff has been upgraded, but we're now waiting on the front end rebuilds, which takes a while.

Initial testing indicates that the kernel is more stable; we're mostly able to unload/reload RTS modules without the kernel freezing.  However, the c1iscey host seems to be oddly problematic and has frozen twice so far on module unloads.  None of the other hosts have frozen on unload (yet), though, so still not clear.

We're now seeing some timing errors between the front ends and daqd, resulting in a "0x4000" status message in the 'C1:DAQ-DC0_*_STATUS' channels.  Part of the problem was an issue with the IRIG-B/GPS receiver timing unit, which I'll log in a separate post.  Another part of the problem was a bug in the symmetricom driver, which has been resolved.  That wasn't the whole problem, though, since we're still seeing timing errors.  Working with Jonathan to resolve.

  13204   Mon Aug 14 16:24:09 2017 gautamUpdateALSFiber ALS

Today, I borrowed the fiber microscope from Johannes and took a look at the fibers coupled to the PDs. The PD labelled "BEAT PD AUX Y" has an end that seems scratched (Attachments #1 and #2). The scratch seems to be on (or at least very close to) the core. The other PD (Attachments #3 and #4) doesn't look very clean either, but at least the area near the core seems undamaged. The two attachments for each PD corresponds to the two available lighting settings on the fiber microscope.

I have not attempted to clean them yet, though I have also borrowed the cleaning supplies to facilitate this from Johannes. I also plan to inspect the ends of all other fiber connections before re-installing them.

Quote:

Last week, we were talking about reviving the Fiber ALS box. Right now, it's not in great shape. Some changes to be made:

  1. Supply power to the PDs (Menlo FPD310) via a power regulator board. The datasheet says the current consumption per PD is 250 mA. So we need 500mA. We have the D1000217 power regulator board available in the lab. It uses the LM2941 and LM2991 power regulator ICs, both of which are rated for 1A output current, so this seems suitable for our purposes. Thoughts?
  2. Install power decoupling capacitors on the PDs.
  3. Clean up the fiber arrangement inside the box.
  4. Install better switches, plus LED indicators.
  5. Cover the box.
  6. Install it in a better way on the PSL table. Thoughts? e.g. can we mount the unit in some electronics rack and route the fibers to the rack? Perhaps the PSL IR and one of the arm fibers are long enough, but the other arm might be tricky.

Previous elog thread about work done on this box: elog11650

 

Attachment 1: IMG_7471.JPG
IMG_7471.JPG
Attachment 2: IMG_7472.JPG
IMG_7472.JPG
Attachment 3: IMG_7473.JPG
IMG_7473.JPG
Attachment 4: IMG_7474.JPG
IMG_7474.JPG
  13203   Mon Aug 14 12:52:33 2017 KiraUpdatePEMtemperature sensor

I didn't realize that the LT1012 needed an additional input to function. I added in +15V and -15V to pins 7 and 4, respectively and placed a 10k resistor and the numbers make more sense now. The voltage showed a negative value, but it became more negative as I heated it up (it's negative due to how a transimpedance amplifier works).

I have attached the new setup and the value it shows (~-3V). It became more negative by about 0.4V, which translates to about a 40K increase in temperature, which makes sense.

In addition, I have attached an updated sketch of the circuit. I will need to do more testing to determine how accurate this is. The next step would be to calculate how much noise there is currently and figure out how to remove this circuit from the breadboard and use a PCB or something like that for final testing in an insulated container.

The reason I chose AD743 initially for the OP amp is because at low frequencies (which is what we are working with), a FET amp such as AD743 will have a low current noise at high impedance, which is what we have in this case. While a FET amp has high voltage noise compared to other OP amps, the current noise becomes more important at high impedance, so it will work better. According to Zach's graphs, the AD743 is best at high impedances, followed by LT1012.

Quote:

Decided to try adding in an OP amp just to see if it would work. Added LT1012 and a 100k resistor to the circuit (I originally wanted to do AD743 as it seems to be the best choice according to Zach's elog here, but it said that they are very precious so I went with LT1012 for testing purposes). When heating it with a heating gun, the output voltage went down by a few 0.01V. The maximum voltage was 0.686V. Similar thing happened when I switched to a 10k resistor, where the maximum was 0.705V and it also went down by a few 0.01V upon heating.

I've attached a few pictures showing the circuit.

 

Attachment 1: IMG_20170814_121131.jpg
IMG_20170814_121131.jpg
Attachment 2: IMG_20170814_121139.jpg
IMG_20170814_121139.jpg
Attachment 3: IMG_20170814_121758~2.jpg
IMG_20170814_121758~2.jpg
  13202   Mon Aug 14 09:49:18 2017 KiraUpdatePEMtemperature sensor

Decided to try adding in an OP amp just to see if it would work. Added LT1012 and a 100k resistor to the circuit (I originally wanted to do AD743 as it seems to be the best choice according to Zach's elog here, but it said that they are very precious so I went with LT1012 for testing purposes). When heating it with a heating gun, the output voltage went down by a few 0.01V. The maximum voltage was 0.686V. Similar thing happened when I switched to a 10k resistor, where the maximum was 0.705V and it also went down by a few 0.01V upon heating.

I've attached a few pictures showing the circuit.

Attachment 1: IMG_20170814_092452.jpg
IMG_20170814_092452.jpg
Attachment 2: IMG_20170814_092513.jpg
IMG_20170814_092513.jpg
  13201   Sun Aug 13 01:35:09 2017 KojiUpdateSUSGlitches stay on MC1

We used to have similar suspension excursion at ETMX. This was the motivation to replace the stand-offs from Al ones to ruby ones. Did the replacement solve the issue at ETMX?

  13200   Sat Aug 12 22:35:22 2017 ranaUpdateSUSGlitches stay on MC1

To add to Gautam's entry: we swapped the cables at the coil driver side (these are the ones that go from coil driver to sat box). In this state, damping is not useable since the MC1 servos would drive MC3.

~70 counts in the sensor means ~70 microns of motion. Since the watchdogs are off and the coil drivers are swapped, this can't be laser beam getting in to the sensors.

WE have to consider that these are some real strain release type events happening in the suspension wire or wire standoff, so may require a vent to inspect and possible repair MC1.

  13199   Sat Aug 12 14:09:36 2017 gautamUpdateSUSGlitches stay on MC1

Even in the switched state, the glitches stayed on MC1.

The coil driver electronics for MC1, upstream of the Satellite box, was what was previously MC3 electronics.

Attachment #1 shows that there were no glitches in MC3 sensor channels (which are now physically connected to what was previously MC1 coil driver electronics).

Attachment #2 shows the second trends for a 12 hour period for MC1 and MC3 sensor channels. The MC3 channels look well behaved, but there are frequent glitches (at least 9 in the last 12 hours indecision) visible in the MC1 channels.

So to recap:

  • We switched MC1 satellite box - but glitch stayed on MC1, so it would seem the Satellite box is not to blame.
  • We shutdown the watchdog and the glitches persisted.
  • We switched the coil driver electronics for MC1, but glitches remained on MC1, and MC3 doesn't show any evidence of glitching. This and the previous bullet point suggest the coil driver electronics are not to blame.
  • For the glitch posted in Attachment #1, I could see the MC-REFL spot moving around on the CCD monitors, so the glitches aren't just a feature in the shadow sensor readout. 

I need to confirm that the output of the coil driver board goes straight to the Sat. Box, but if there are no intermediate elements, the problem is either in the cable from coil driver to sat. box, or downstream of the Satellite box - i.e. vacuum feedthroughs or the suspension itself? The size of the glitches is roughly the same in all 4 face channels (~60-80cts pp).

Quote:

About 30mins ago, I saw another glitch on MC1 - this happened while the Watchdog was shutdown.

In order to further narrow down the cause of the glitch, we switched the Coil Driver Board --> Satellite box DB(15?) connectors on the coil drivers between MC1 and MC3 coil driver boards. I also changed the static PIT/YAW bias voltages to MC1 and MC3 such that MC-REFL is now approximately back to the center of the CCD monitor.

 


GV addendum 14 Aug 2017, 10.30am: Attachment #3 shows the second trend for the MC sensor channels over the weekend. While there were many on Saturday, it seems that Sunday was quieter.

Attachment 1: MC1_glitch.png
MC1_glitch.png
Attachment 2: MC_12hr_trend.png
MC_12hr_trend.png
Attachment 3: MC1_glitches_intermittent.png
MC1_glitches_intermittent.png
  13198   Fri Aug 11 19:34:49 2017 JamieUpdateCDSCDS final bits status update

So it appears we now have full frames and second, minute, and minute_raw trends.

We are still not able to raise test points with daqd_rcv (e.g. the NDS1 server), which is why dataviewer and nds2-client can't get test points on their own.

We were not able to add the EDCU (EPICS client) channels without daqd_fw crashing.

We have a new kernel image that's supposed to solve the module unload instability issue.  In order to try it we'll need to restart the entire system, though, so I'll do that on Monday morning.

I've got the CDS guys investigating the test point and EDCU issues, but we won't get any action on that until next week.

Quote:

Remaining unresolved issues:

  • IFO needs to be fully locked to make sure ALL components of all models are working.
  • The remaining red status lights are from the "FB NET" diagnostics, which are reflecting a missing status bit from the front end processes due to the fact that they were compiled with an earlier RCG version (3.0.3) than the mx_streams were (3.3+/trunk).  There will be a new release of the RTS soon, at which point we'll compile everything from the same version, which should get us all green again.
  • The entire system has been fully modernized, to the target CDS reference OS (Debian jessie) and more recent RCG versions.  The management of the various RTS components, both on the front ends and on fb, have as much as possible been updated to use the modern management tools (e.g. systemd, udev, etc.).  These changes need to be documented.  In particular...
  • The fb daqd process has been split into three separate components, a configuration that mirrors what is done at the sites and appears to be more stable: The "target" directory for all of these components is now:
    • daqd_dc: data concentrator (receives data from front ends)
    • daqd_fw: receives frames from dc and writes out full frames and second/minute trends
    • daqd_rcv: NDS1 server (raises test points and receives archive data from frames from 'nds' process)
    The "target" directory for all of these new components is:
    • /opt/rtcds/caltech/c1/target/daqd
    All of these processes are now managed under systemd supervision on fb, meaning the daqd restart procedure has changed.  This needs to be simplified and clarified.
  • Second trend frames are being written, but for some reason they're not accessible over NDS.
  • Have not had a chance to verify minute trend and raw minute trend writing yet.  Needs to be confirmed.
  • Get wiper script working on new fb.
  • Front end RTS kernel will occaissionally crash when the RTS modules are unloaded.  Keith Thorne apparently has a kernel version with a different set of patches from Gerrit Kuhn that does not have this problem.  Keith's kernel needs to be packaged and installed in the front end diskless root.
  • The models accessing the dolphin shared memory will ALL crash when one of the front end hosts on the dolphin network goes away.  This results in a boot fest of all the dolphin-enabled hosts.  Need to figure out what's going on there.
  • The RCG settings snapshotting has changed significantly in later RCG versions.  We need to make sure that all burt backup type stuff is still working correctly.
  • Restoration of /frames from old fb SCSI RAID?
  • Backup of entirety of fb1, including fb1 root (/) and front end diskless root (/diskless)
  • Full documentation of rebuild procedure from Jamie's notes.
  13197   Fri Aug 11 18:53:35 2017 gautamUpdateCDSSlow EPICS channels -> Frames re-enabled
Quote:

Seems like something has failed after I did this - full frames are no longer on Aug 10 being written since ~2.30pm PDT. I found out when I tried to download some of the free-swinging MC1 data.

To clarify, I logged into fb1, and ran sudo systemctl restart daqd_*. The only change I made was to uncomment the line quoted below in the master file.

Looking at the log using systemctl, I see the following (I just tried restarting the daqd processes again):

Aug 11 00:00:31 fb1 daqd_fw[16149]: LDASUnexpected::unexpected: Caught unexpected exception      "This is a bug. Please log an LDAS problem report including this message.
Aug 11 00:00:31 fb1 daqd_fw[16149]: daqd_fw: LDASUnexpected.cc:131: static void LDASTools::Error::LDASUnexpected::unexpected(): Assertion `false' failed.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service: main process exited, code=killed, status=6/ABRT
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Aug 11 00:00:32 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Aug 11 00:00:32 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.

Oddly, I am able to access second trends for the same channels from the past which will be useful for the MC1 debugging). Not sure whats going on.


The live data grabbing using cdsutils still seems to be working though - so I've kicked MC1 again, and am grabbing 2 hours of data live on Pianosa.

So we tried this again with a fresh build of daqd_fw, and it still fails.  The error message is pointing to an underlying bug in the framecpp library ("LDASTools"), which may be tricky to solve.  I'm rustling the appropriate bushes...

  13196   Fri Aug 11 17:36:47 2017 gautamUpdateSUSMC1 <--> MC3

About 30mins ago, I saw another glitch on MC1 - this happened while the Watchdog was shutdown.

In order to further narrow down the cause of the glitch, we switched the Coil Driver Board --> Satellite box DB(15?) connectors on the coil drivers between MC1 and MC3 coil driver boards. I also changed the static PIT/YAW bias voltages to MC1 and MC3 such that MC-REFL is now approximately back to the center of the CCD monitor.

 

Attachment 1: MC1_glitch_watchdog_shutdown.png
MC1_glitch_watchdog_shutdown.png
  13195   Fri Aug 11 12:32:46 2017 gautamUpdateSUSMC1 glitches debugging

Attachment #1: Free swinging sensor spectra. I havent done any peak fitting but the locations of the resonances seem consistent with where we expect them to be.

The MC_REFL spot appears to not have shifted significantly (so slow bias voltages are probably not to blame). Now I have to look at trend data to see if there is any evidence of glitching.

I'm not sure I understand the input matrix though - the matrix elements would have me believe that the sensing of POS in UL is ~5x stronger than in UR and LL, but the peak heights don't back that up.

Attachment #3: Second trend over 5hours (since frame writing was re-enabled this morning). Note that MC1 is still free-swinging but there is no evidence of steps of ~30cts which have been observed some days ago. Also, from my observations yesterday, MC1 glitched multiple times over a few hours timescale. More data will have to be looked at, but as things stand, Hypothesis #3 below looks the best.

Quote:
 

Some possible scenarios (assuming the free swinging spectra look alright and the various resonances are where we expect them to be):

  1. With the watchdog shutdown, the PIT/YAW bias voltages still goto the coil (low-passed by 4 poles @1Hz). So if the glitching happens in this path, we should see it in both the shadow sensors and the DC spot positions on the WFS.
  2. If the glitching happens in the shadow sensor readout electronics/cabling, we should see it in the shadow sensor channels, but NOT in the DC spot positions on the WFS (as the watchdog is shutdown, so there should be no actuation to the coils based on OSEM signals).
  3. If we don't see any glitches in WFS spot positions or shadow sensors, then it is indicative of the problem being in the coil driver board / dewhitening board/anti-aliasing board.
  4. I am discounting the problem being in the Satellite box, as we have switched around the MC1 satellite box multiple times - the glitches remain on MC1 and don't follow a Satellite Box. Of course there is the possibility that the cabling from 1X5/1X6 to the Satellite box is bad.

 

Attachment 1: MC1_freeswinging.pdf
MC1_freeswinging.pdf
Attachment 2: MC1_inmatrix.png
MC1_inmatrix.png
Attachment 3: MC1_sensors.png
MC1_sensors.png
  13194   Fri Aug 11 12:27:25 2017 KiraUpdatePEMtemperature sensor

Used AD592CNZ and AD586 (5V output) to create a circuit that works and is responsive to temperature changes. At room temp, using ~1K resistor, it showed ~0.3V across it, as expected. The voltage went up when we heated it with a heating gun. Next step will be to add in an OP amp and design some experiments to check to see how accurate it is. Thanks to Gautam for helping me with it!

I have attached the working circuit and a close up of the connections.

Attachment 1: IMG_20170811_121608.jpg
IMG_20170811_121608.jpg
Attachment 2: IMG_20170811_121619.jpg
IMG_20170811_121619.jpg
  13193   Fri Aug 11 11:56:02 2017 ranaUpdatePEMtemperature sensor

Might as well order several of a few different varieties today. Its good to have some extra in stock; we don't always want to have to wait days for parts to show up. If you give Steve a list of parts to buy he can order them today or Monday.

There should also be some precision 5V sources (e.g. AD586) that you can try.

  13192   Fri Aug 11 11:14:24 2017 gautamUpdateCDSSlow EPICS channels -> Frames re-enabled

I commented out the line pertaining to C0EDCU again, now full frames are being written again.

But we no longer have access to the slow EPICS records.

I am not sure what the failure mode is here - In the master file, there is a line that says the EDCU list "*MUST* COME *AFTER* ALL OTHER FAST INI DEFINITIONS" which it does. But there are a bunch of lines that are testpoint lists after this EDCU line. I wonder if that is the problem?

Quote:

Seems like something has failed after I did this - full frames are no longer on Aug 10 being written since ~2.30pm PDT. I found out when I tried to download some of the free-swinging MC1 data.

 

  13191   Fri Aug 11 10:48:39 2017 KiraUpdatePEMtemperature sensor

Quick update: we actually have AD587KRZ and AD592, so we could start by using that and seeing how it works.

  13190   Fri Aug 11 10:27:49 2017 KiraUpdatePEMtemperature sensor

Since there seems to be little difference between AD590 and AD592, I guess we could just go with the AD590. The temperature noise spectrum in the first graph are for the AD590, so if we want to reproduce those results, we should use AD590.

For the AD581/AD587, we could go with a few varieties that have the least output voltage drift, although I am not sure what precision we will need. So maybe we could try AD587U and AD581L. We could also try AD587K and AD581K and see if those work as well.

We will also need to calibrate the sensor, as it takes an input of 5V, but the AD581/AD587 provides 10V, which will give about a 1 degree error according to the datasheet. It does state that this is only a calibration error, so it shouldn't be too much of an issue.

I will figure out the packaging once I construct the sensor and verify that it works. Maybe we could use a box similar to the existing sensor, but it depends on the size of the finished circuit.

  13189   Fri Aug 11 00:10:03 2017 gautamUpdateCDSSlow EPICS channels -> Frames re-enabled

Seems like something has failed after I did this - full frames are no longer on Aug 10 being written since ~2.30pm PDT. I found out when I tried to download some of the free-swinging MC1 data.

To clarify, I logged into fb1, and ran sudo systemctl restart daqd_*. The only change I made was to uncomment the line quoted below in the master file.

Looking at the log using systemctl, I see the following (I just tried restarting the daqd processes again):

Aug 11 00:00:31 fb1 daqd_fw[16149]: LDASUnexpected::unexpected: Caught unexpected exception      "This is a bug. Please log an LDAS problem report including this message.
Aug 11 00:00:31 fb1 daqd_fw[16149]: daqd_fw: LDASUnexpected.cc:131: static void LDASTools::Error::LDASUnexpected::unexpected(): Assertion `false' failed.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service: main process exited, code=killed, status=6/ABRT
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Aug 11 00:00:32 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Aug 11 00:00:32 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Aug 11 00:00:32 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Aug 11 00:00:32 fb1 systemd[1]: Unit daqd_fw.service entered failed state.

Oddly, I am able to access second trends for the same channels from the past which will be useful for the MC1 debugging). Not sure whats going on.


The live data grabbing using cdsutils still seems to be working though - so I've kicked MC1 again, and am grabbing 2 hours of data live on Pianosa.

Quote:

I went into /opt/rtcds/caltech/c1/target/daqd, opened the master file, and uncommented the line with C0EDCU.ini (this is the file in which all the slow machine channels are defined). So now I am able to access, for example, the c1vac1 channels.

The location of the master file is no longer in /opt/rtcds/caltech/c1/target/fb, but is in the above mentioned directory instead. This is part of the new daqd paradigm in which separate processes are handling the data transfer between FEs and FB, and the actual frame-writing. Jamie will explain this more when he summarizes the CDS revamp.

It looks like trend data is also available for these newly enabled channels, but thus far, I've only checked second trends. I will update with a more exhaustive check later in the evening.

So, the two major pending problems (that I can think of) are:

  1. Inability to unload models cleanly
  2. Inability of dataviewer (and cdsutils) to open testpoints.

Apart from this, dataviewer frequently hangs on Donatella at startup. I used ipcs -a | grep 0x | awk '{printf( "-Q %s ", $1 )}' | xargs ipcrm to remove all the extra messages in the dataviewer queue.


Restarting the daqd processes on fb1 using Jamie's instructions from earlier in this thread works - but the mx_stream processes do not seem to come back automatically on c1lsc, c1sus and c1ioo (reasons unknown). I've made a copy of the mxstreamrestart.sh script with the new mxstream restart commands, called mxstreamrestart_debian.sh, which lives in /opt/rtcds/caltech/c1/scripts/cds. I've also modified the CDS overview MEDM screen such that the "mxstream restart" calls this modified script. For now, this requires you to enter the controls password for each machine. I don't know what is a secure way to do it otherwise, but I recall not having to do this in the past with the old mxstreamrestart.sh script.

 

  13188   Thu Aug 10 21:22:06 2017 ranaSummaryPEMtemperature sensor
  • Should we use the AD590 or the AD592? They're sort of the same, but have slightly different packages and specs.
  • Given the sensitivity of the AD590 to power supply drift, I would recommend using a precision voltage reference like the AD581 or AD587. Take a look at the datasheets and order a few different varieties so we can see what works best for us. I believe that the voltage regulators have too much drift to use for precision temperature control.
  • The AD590 datasheet has a few example circuits showing how we can subtract off the offset which comes the 1 uA/K coefficient of the AD590 (i.e. we would have 295 uA at room temperature, trying to stabilize to +/- 0.1 K)
  • For the first prototypes its fine to use some solderable protoboard assembly, but we will eventually have to figure out how to package this thing and interface it with our Acromag slow controls system.

also, I've attached some temperature noise spectra from the LISA group at the AEI in Hannover. It will be interesting to see if we get the same results.

Attachment 1: tempsensnoise2.pdf
tempsensnoise2.pdf
Attachment 2: tempnoise_final.pdf
tempnoise_final.pdf
  13187   Thu Aug 10 21:01:43 2017 gautamUpdateSUSMC1 glitches debugging

I have squished cables in all the places I can think of - but MC1 has been glitching regularly today. Before starting to pull electronics out, I am going to attempt a more systematic debugging in the hope I can localize the cause.

To this end, I've disabled the MC autolocker, and have shutdown the MC1 watchdog. I plan to leave it in this state overnight. From this, I hope to look at the free-swinging optic spectra to see that this isn't a symptom of something funky with the suspension itself.

Some possible scenarios (assuming the free swinging spectra look alright and the various resonances are where we expect them to be):

  1. With the watchdog shutdown, the PIT/YAW bias voltages still goto the coil (low-passed by 4 poles @1Hz). So if the glitching happens in this path, we should see it in both the shadow sensors and the DC spot positions on the WFS.
  2. If the glitching happens in the shadow sensor readout electronics/cabling, we should see it in the shadow sensor channels, but NOT in the DC spot positions on the WFS (as the watchdog is shutdown, so there should be no actuation to the coils based on OSEM signals).
  3. If we don't see any glitches in WFS spot positions or shadow sensors, then it is indicative of the problem being in the coil driver board / dewhitening board/anti-aliasing board.
  4. I am discounting the problem being in the Satellite box, as we have switched around the MC1 satellite box multiple times - the glitches remain on MC1 and don't follow a Satellite Box. Of course there is the possibility that the cabling from 1X5/1X6 to the Satellite box is bad.

MC1 has been in a glitchy mood today, with large (MC-REFL spot shifts by ~1 beam diameter on the CCD monitor) glitches happening ~every 2-3 hours. Hopefully it hasn't gone into an extended quiet period. For reference, I've attached the screen-grab of the MC-QUAD and MC-REFL as they are now.


GV 9.20PM: Just to make sure of good SNR in measuring the pendulum eigenfreqs, I ran /opt/rtcds/caltech/c1/scripts/SUS/freeswing MC1 in a terminal . The result looked rather violent on the camera but its already settling down. The terminal output:

The following optics were kicked:
MC1
Thu Aug 10 21:21:24 PDT 2017
1186460502
Quote:

Happened again just now, although the characteristics of the glitch are very different from the previous post, its less abrupt. Only actuation on MC1 at this point was local damping.

 

Attachment 1: MC_QUAD_10AUG2017.jpg
MC_QUAD_10AUG2017.jpg
Attachment 2: MCR_10AUG2017.jpg
MCR_10AUG2017.jpg
  13186   Thu Aug 10 15:45:34 2017 SteveUpdateVACaccidental vent to 17 Torr

Finally we got the cold cathode gauge working. IFO pressure 7e-6 Torr-it, vacuum normal valve configuration with all 4 ion pump gate valves closed at ~ 9e-6 Torr.  The cryo pump was also pumped yesterday to remove the accumulated outgassing build up.

The accidental vent was my mistake; made when I was replacing the battery pack of the UPS. The installed pack measured 51 V without any load. The "replace battery" warning light did not go out after the batteries were replaced. 

I then mistakenly and repeatedly pushed the "test" button to reset this, but I did not wait long enough for the batterries to get fully charged. The test put the full load on the new batteries and their charging condition got worse. I made the mistake when trying to put the load from the battery to online and pushed "O" so the power was cut and the computer rebooted to the all off condition. On top of this, I disconnected the wrong V1 cable to close V1.  As the computer rebooted it's interlock closed V1 at 17 Torr.

Never hit O on the Vacuum UPS !

Note: the " all off " configuration should be all valves closed ! This should be fixed now.

In case of  emergency  you can close V1 with disconnecting it's actuating power as shown on Atm3 if you have peumatic pressure 60 PSI 

Attachment 1: Mag_UPS.jpg
Mag_UPS.jpg
Attachment 2: UPSbatteryPack.jpg
UPSbatteryPack.jpg
Attachment 3: closing_V1.jpg
closing_V1.jpg
  13185   Thu Aug 10 14:25:52 2017 gautamUpdateCDSSlow EPICS channels -> Frames re-enabled

I went into /opt/rtcds/caltech/c1/target/daqd, opened the master file, and uncommented the line with C0EDCU.ini (this is the file in which all the slow machine channels are defined). So now I am able to access, for example, the c1vac1 channels.

The location of the master file is no longer in /opt/rtcds/caltech/c1/target/fb, but is in the above mentioned directory instead. This is part of the new daqd paradigm in which separate processes are handling the data transfer between FEs and FB, and the actual frame-writing. Jamie will explain this more when he summarizes the CDS revamp.

It looks like trend data is also available for these newly enabled channels, but thus far, I've only checked second trends. I will update with a more exhaustive check later in the evening.

So, the two major pending problems (that I can think of) are:

  1. Inability to unload models cleanly
  2. Inability of dataviewer (and cdsutils) to open testpoints.

Apart from this, dataviewer frequently hangs on Donatella at startup. I used ipcs -a | grep 0x | awk '{printf( "-Q %s ", $1 )}' | xargs ipcrm to remove all the extra messages in the dataviewer queue.


Restarting the daqd processes on fb1 using Jamie's instructions from earlier in this thread works - but the mx_stream processes do not seem to come back automatically on c1lsc, c1sus and c1ioo (reasons unknown). I've made a copy of the mxstreamrestart.sh script with the new mxstream restart commands, called mxstreamrestart_debian.sh, which lives in /opt/rtcds/caltech/c1/scripts/cds. I've also modified the CDS overview MEDM screen such that the "mxstream restart" calls this modified script. For now, this requires you to enter the controls password for each machine. I don't know what is a secure way to do it otherwise, but I recall not having to do this in the past with the old mxstreamrestart.sh script.

  13184   Thu Aug 10 14:14:17 2017 KiraUpdatePEMpreviously built temp sensor

I decided to see what was inside the sensor that had been previously made. According to elog 1102, the temperature sensor is LM34, the specs of which can be found here:

http://www.ti.com/lit/ds/symlink/lm34.pdf

The wiring of this sensor confused me, as it appears that the +Vs end (white) connects to the input, but both the ground (left) and the Vout (middle) pins are connected to the box itself. I don't see how the signal can be read.

Attachment 1: IMG_20170810_112315.jpg
IMG_20170810_112315.jpg
  13183   Thu Aug 10 14:13:23 2017 KiraSummaryPEMtemperature sensor

Goal is to build a temperature sensor accurate to 1 mK. Schematic is shown below. This does not take into account the DC gain that occurs.

Parts that would be used for this: LM317 regulator, AD592 temperature transducer, OP amp (low input noise and high impedance), 100K (or maybe 10k) resistor. This is what is currently proposed, but the exact parts we use could be changed to better fit the sensor. The resistor and the OP amp will be decided depending on the output of the AD592.

Once this is built, I would like to create a few copies of it and put them into an insulated container and measure the output from each one. This would allow us to calculate the temperature noise of the circuit, as we can take out the variations due to temperature changes inside the container by comparing the outputs.

I can also model the noise in the circuit to see how much noise there is before building it. There are three terms to the noise that we have, and we need to decide which one dominates at low frequencies.

Our final goal is to create an additional circuit that could cancel out the DC gain. I have attached an additional schematic proposed by Rana that would help with this issue. I will leave this second half for when the first part works.

Attachment 1: IMG_20170810_121637~2.jpg
IMG_20170810_121637~2.jpg
Attachment 2: IMG_20170810_134422~2.jpg
IMG_20170810_134422~2.jpg
  13182   Thu Aug 10 09:31:57 2017 SteveUpdateSUSITMX sensor voltage

There must be some bad connection

Quote:

Somewhere between CDS model restarts and the IFO venting, ITMX got stuck.

I shook it loose using the usual bias slider technique. It appears to be free now, I was able to lock the green beam on a TEM00 mode without touching the green input pointing. The ITMX Oplev spot has also returned to within its MEDM display bounds.

 

Attachment 1: 9daysITMX.png
9daysITMX.png
Attachment 2: vacGlitchITMX.png
vacGlitchITMX.png
  13181   Thu Aug 10 09:10:55 2017 steveUpdateGeneraldataviewer is recovering

It can look back 7 days trends now. There is still no vacuum channels. I can bring back the channels through the restore directory, but there are no data.

Attachment 1: 7dm.png
7dm.png
  13180   Wed Aug 9 19:21:18 2017 gautamUpdateALSALS recovery

Summary:

Between frequent MC1 excursions, I worked on ALS recovery today. Attachment #1 shows the out-of-loop ALS noise as of today evening (taken with arms locked to IR) - I have yet to check loop shapes of the ALS servos, looks like there is some tuning to be done.

On the PSL table:

  • First, I locked the arms to IR, ran the dither alignment servos to maximize transmission.
  • I used the IR beat PDs to make sure a beat existed, at approximately.
  • Then I used a scope to monitor the green beat, and tweaked steering mirror alignment until the beat amplitude was maximized. I was able to improve the X arm beat amplitude, which Koji and Naomi had tweaked last week, by ~factor of 2, and Y arm by ~factor of 10.
  • I used the DC outputs of the BBPDs to center the beam onto the PD.
  • Currently, the beat notes have amplitudes of ~-40dBm on the scopes in the control room (there are various couplers/amplifiers in the path so I am not sure what beatnote amplitude this translates to at the BBPD output). I have yet to do a thorough power budget, but I have in my mind that they used to be ~-30dBm. To be investigated.
  • Removed the fiber beat PD 1U chassis unit from the PSL table for further work. The fibers have been capped and remain on the PSL table. Cleaned the NW corner of the PSL table up a bit.

To do:

  • Optimization of the input pointing of the green beam for X (with PZTs) and Y (manual) arms.
  • ALS PDH servo loop measurement. Attachment #1 suggests some loop gain adjustment is required for both arms (although the hump centered around ~70Hz seem to be coming from the IR lock).
  • Power budgeting on the PSL table to compare to previous such efforts.

Note: Some of the ALS scripts are suffering from the recent inablilty of cdsutils to pull up testpoints (e.g. the script that is used to set the UGFs of the phase tracker servo). The workaround is to use DTT to open the test points first (just grab 0.1s time series for all channels of interest). Then the cdsutils scripts can read the required channels (but you have to keep the DTT open).

Attachment 1: ALS_oolSpec.pdf
ALS_oolSpec.pdf
  13179   Wed Aug 9 16:34:46 2017 ranaUpdateVACVacuum Document recovered

Steve and I found the previous draft of the 40m Vacuum Document. Someone in 2015 had browsed into the Docs history and then saved the old 2013 version as the current one.

We restored the version from 2014 which has all of Steve's edits. I have put that version (which is now the working copy) into the DCC:  https://dcc.ligo.org/E1500239.

The latest version is in our Google Docs place as usual. Steve is going to have a draft ready for us to ready be Tuesday, so please take a look then and we can discuss what needs doing at next Wednesday's 40m meeting.

  13178   Wed Aug 9 15:15:47 2017 gautamUpdateSUSMC1 glitches return

Happened again just now, although the characteristics of the glitch are very different from the previous post, its less abrupt. Only actuation on MC1 at this point was local damping.

Attachment 1: MC1_glitch.png
MC1_glitch.png
  13177   Wed Aug 9 12:35:47 2017 gautamUpdateALSFiber ALS

Last week, we were talking about reviving the Fiber ALS box. Right now, it's not in great shape. Some changes to be made:

  1. Supply power to the PDs (Menlo FPD310) via a power regulator board. The datasheet says the current consumption per PD is 250 mA. So we need 500mA. We have the D1000217 power regulator board available in the lab. It uses the LM2941 and LM2991 power regulator ICs, both of which are rated for 1A output current, so this seems suitable for our purposes. Thoughts?
  2. Install power decoupling capacitors on the PDs.
  3. Clean up the fiber arrangement inside the box.
  4. Install better switches, plus LED indicators.
  5. Cover the box.
  6. Install it in a better way on the PSL table. Thoughts? e.g. can we mount the unit in some electronics rack and route the fibers to the rack? Perhaps the PSL IR and one of the arm fibers are long enough, but the other arm might be tricky.

Previous elog thread about work done on this box: elog11650

Attachment 1: IMG_3942.JPG
IMG_3942.JPG
  13176   Wed Aug 9 12:05:57 2017 ranaUpdateElectronicsdata archiving

This kind of data fitting and analysis is really useful. We should figure out a way to archive it. Perhaps the data files and fitting stuff can be put into GIT in some smart way? The fit results can be added to the 40m MC electronics DCC tree. Then the links can be added to this elog.

  13175   Wed Aug 9 11:59:51 2017 SteveUpdateVACunintended pump down at vacuum normal

pd80b has reached Vac Normal. IFO pressure 0.5 mTorr

We need our vauum channels back in dataviewer.

Quote:

Pumpdown stopped for over night  at ~  1 Torr

The roughing line disconnected.  Valves condition indicator  "moving " means that it is closed and it's cable disconnected so it can not move.

The RGA is off and VM1 is stuck.

Quote:

IFO pressure 2 Torr,    PSL  shutter closed.  I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses at atm.

The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later.....

Do not plan on using the interferrometer tonight.  I will complete the pumpdown tomorrow morning.

 

 

 

Attachment 1: pdc.png
pdc.png
Attachment 2: pd80b@vacnormal.png
pd80b@vacnormal.png
  13174   Wed Aug 9 11:33:49 2017 gautamUpdateElectronicsMC2 de-whitening

Summary:

The analog de-whitening filters for MC2 are different from those on the other optics (i.e. ITMs and ETMs). They have one complex pole pair @7Hz, Q~sqrt(2), one complex zero pair @50Hz, Q~sqrt(2), one real pole at 2.5kHz, and one real zero @250Hz (with a DC gain of 10dB).

Details:

I took the opportunity last night to measure all 4 de-whitening channel TFs. Measurements and overlaid LISO fits are seen in Attachment #1. 

The motivation behind this investigation was that last week, I was unable to lock the IMC to one of the arms. In the past, this has been done simply by routing the control signal of the appropriate arm filter bank (e.g. C1:LSC-YARM_OUT) to MC2 instead of ETMY via the LSC output matrix (if the matrix element to ETMY is 1, the matrix element to MC2 is -1).

Looking at the coil output filter banks on the MC2 suspension MEDM screen (see Attachment #2), the positions of filters in the filter banks is different from that on the other optics. In general, the BIO outputs of the DAC are wired such that disengaging FM9 on the MEDM screen engages the analog de-whitening path. FM10 then has the inverse of the de-whitening filter, such that the overall TF from DAC to optic is unity. But on MC2, these filters occupy FM7 and FM8, and FM9 was originally a 28Hz Elliptic Low-pass filter.

So presumably, I was unable to lock the IMC to an arm because for either configuration of FM9 (ON or OFF), the signal to the optic was being aggressively low-passed. To test this hypothesis, I simply copied the 28Hz elliptic to FM6, put a gain of 1 on FM9, left it engaged (so that the analog path TF is just flat with gain x3), and tried locking the IMC to the arm again - I was successful. See Attachment #3 for comparison of the control signal spectra of the X-arm control signal, with the IMC locked to the Y-arm cavity.

In this test, I also confirmed that toggling FM9 in the coil output filter banks actually switches the analog path on the de-whitening boards.

Since I now have the measurements for individual channels, I am going to re-configure the filter arrangement on MC2 to mirror that on the other optics. 


Unrelated to this work: the de-whitening boards used for MC1 and MC3 are D000316, as opposed to D000183 used for all other SOS optics. From the D000316 schematic, it looks like the signals from the AI board are routed to this board via the backplane. I will try squishing this backplane connector in the hope it helps with the glitching MC1 suspension.


GV Aug 13 11:45pm - I've made a DCC page for the MC2 dewhitening board. For now, it has the data from this measurement, but if/when we modify the filter shape, we can keep track of it on this page (for MC2 - for the other suspensions, there are other pages). 

Attachment 1: MC2deWhites.pdf
MC2deWhites.pdf
Attachment 2: MC2Coils.png
MC2Coils.png
Attachment 3: MC2stab.pdf
MC2stab.pdf
  13173   Tue Aug 8 20:48:06 2017 gautamUpdateSUSITMX stuck

Somewhere between CDS model restarts and the IFO venting, ITMX got stuck.

I shook it loose using the usual bias slider technique. It appears to be free now, I was able to lock the green beam on a TEM00 mode without touching the green input pointing. The ITMX Oplev spot has also returned to within its MEDM display bounds.

  13172   Tue Aug 8 17:44:11 2017 SteveUpdateVACunintended pump down

Pumpdown stopped for over night  at ~  1 Torr

The roughing line disconnected.  Valves condition indicator  "moving " means that it is closed and it's cable disconnected so it can not move.

The RGA is off and VM1 is stuck.

Quote:

IFO pressure 2 Torr,    PSL  shutter closed.  I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses at atm.

The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later.....

Do not plan on using the interferrometer tonight.  I will complete the pumpdown tomorrow morning.

 

 

Attachment 1: stopped_pumping.png
stopped_pumping.png
  13171   Tue Aug 8 17:04:26 2017 SteveUpdateVACunintended pump down

IFO pressure 2 Torr,    PSL  shutter closed.  I'm pumping down with 2 roughing pumps with ion pump gate valves open and annulosses at atm.

The vacuum envelope was vented to 17 Torr while I was replacing the USP battery stack. More about this later.....

Do not plan on using the interferrometer tonight.  I will complete the pumpdown tomorrow morning.

 

Attachment 1: pumping_down.png
pumping_down.png
  13170   Mon Aug 7 22:50:57 2017 KojiUpdateGeneralNew wifi router for the GC network installed

I have replaced the old 11n wifi router (CISCO / Linksys) for the GC network with a new one with 11ac technology.

The new one is a 3band wifi router. Thus it has one 2.4GHz (11n) SSID and two 5GHz (11ac) SSIDs. All these have been set to be hidden. Just come to the 40m and find the necessary info for the connection.

Note that the user id / password for the admin tool have been changed from the default values.

  13169   Mon Aug 7 16:00:41 2017 ranaUpdateGeneralBilinear noise coupling

These are not the angular test parameters that we're looking for:

 recall that what we want is the low frequency beam spot variations and the feedback to be limited to a small high frequency band.

e.g. only inject noise at 40-50 Hz, not loud enough to find at 2x the injected frequency.

It should NOT be the case that the high frequency injected noise be dominating the RMS.

The coupling should be ~1e-3; some combination of beam spot mis-centering and beam spot motion.

  13168   Sat Aug 5 11:04:07 2017 gautamUpdateSUSMC1 glitches return

See Attachment #1, which is full (2048Hz) data for a 3 minute stretch around when I saw the MC1 glitch. At the time of the glitch, WFS loops were disabled, so the only actuation on MC1 was via the local damping loops. The oscillations in the MC2 channels are the autolocker turning on the MC2 length tickle.

Nikhil and I tried the usual techniques of squishing cables at the satellite box, and also at 1X4/1X5, but the glitching persists. I will try and localize the problem this weekend. This thread details investigations the last time something like this happened. In the past, I was able to fix this kind of glitching by replacing the (high speed) current buffer IC LM6321M. These are present in a two places: Satellite box (for the shadow sensor LED current drive), and on the coil driver boards. I think we can rule out the slow machine ADCs that supply the static PIT and YAW bias voltages to the optic, as that path is low-passed with a 4th order filter @1Hz, while the glitches that show up in the OSEM sensor channels do not appear to be low-passed, as seen in the zoomed in view of the glitch in Attachment #2 (but there is an LM6321 in this path as well).

Attachment 1: MC1_glitch_Aug42017.png
MC1_glitch_Aug42017.png
Attachment 2: MC1_glitch_zoomed.png
MC1_glitch_zoomed.png
  13167   Fri Aug 4 18:25:15 2017 gautamUpdateGeneralBilinear noise coupling

[Nikhil, gautam]

We repeated the test that EricQ detailed here today. We have downloaded ~10min of data (between GPS times 11885925523 - 11885926117), and Nikhil will analyze it.

Attachment 1: bilinearTest.pdf
bilinearTest.pdf
  13166   Fri Aug 4 09:07:28 2017 ranaUpdateCDSCDS system essentially NOT fully recovered

Tried getting trends with dataviewer just now since Jamie re-enabled the minute_raw frame writing yesterday. Unable to get trends still:

Connecting to NDS Server fb1 (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
unknown error returned from daq_send()T0=17-08-04-08-02-22; Length=28800 (s)
No data output.

  13165   Thu Aug 3 20:15:11 2017 JamieUpdateCDSdataviewer can not raise test points

For some reason dataviewer is not able to raise test points with the new daqd setup, even though dtt can.  If you raise a test point with dtt then dataviewer can show the data fine.

It's unclear to me why this would be the case.  It might be that all the versions of dataviewer on the workstations are too old??  I'll look into it tomorrow to see if I can figure out what's going on.

  13164   Thu Aug 3 19:46:27 2017 JamieUpdateCDSnew daqd restart procedure

This is the daqd restart procedure:

$ ssh fb1 sudo systemctl restart daqd_*

That will restart all of the daqd services (daqd_dc, daqd_fw, daqd_rcv).

The front end mx_stream processes should all auto-restart after the daqd_dc comes back up.  If they don't (models show "0x2bad" on DC0_*_STATUS) then you can execute the following to restart the mx_stream process on the front end:

$ ssh c1<host> sudo systemctl restart mx_stream

 

 

ELOG V3.1.3-