40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 292 of 326  Not logged in ELOG logo
ID Date Author Typeup Category Subject
  14337   Mon Dec 10 12:11:28 2018 aaronUpdateOMCAligning the OMC

I did some ray tracing and determined that the aux beam will enter the OMC after losing some power in reflection on OMPO (couldn't find this spec on the wiki, I remember something like 90-10 or 50-50) and the SRM (R~0.9), and then transmission through OMPO. This gives us something like 8%-23% of the aux light going to the OMC, depending on the OMPO transmission. This elog tells me the aux power before the recombination BS is ~37mW, ~3.7mW onto SRM, which is consistent with the OMPO being 90-10, and would mean the aux power onto the OMC is ~3mW, plenty for aligning into the OMC.

Since the dewhitening board I'd intended to use isn't working (see elog) , I'm gong to scan the OMC length with a function generator while adjusting the alignment by hand, as was briefly attempted during the last vent.

I couldn't identify a PD on the AP table that was the one I had used during the last vent, I suspect I coopted the very same PD for the arm loss measurements. It is a PDA520, which has a large (100mm^2) area so I've repurposed it again to catch the OMC prompt reflection during the mode scans. I've mounted it approximately where I expect the refl beam to exit the AS chamber.

I brought over the cart that usually lives at 1X1 to help me organize materials near the OMC chamber for opening.

I replaced the banana connectors we'd been using to send HV to the HV driver with soldered wires going to the final locking connector only, so now the 150V is on a safe cable.

I powered up the DCPD sat box and again confirmed that it's working. I sent a 500Hz sine wave through the sat box and confirmed that I can see the signal in the DCPD channels I've defined in cds. I gave the TT and OMC-L PZT channels bad assignments on the ADC (right now, what reads as 'OMC_PZT_MON' is actually the unfiltered output from the sat box, while the DCPD channels are for the filtered outputs of the box), because the way the signals are grouped on the cables I can't attach all of them at once. For this vent, I'll only really need the DCPD outputs, and since I have confirmed that I can read out both of those I'll fix up the HV driver mon channels later.

Attachment 1: B9DCF55F-1355-410C-8A29-EE45D43A56A4.jpeg
B9DCF55F-1355-410C-8A29-EE45D43A56A4.jpeg
  14338   Mon Dec 10 12:29:05 2018 aaronUpdateOMCOMC channels

I kept having trouble keeping the power LEDs on the dewhitening board 'on'. I did the following:

1. I noticed that the dewhitening board was drawing a lot of current (>500mA), so I initially thought that the indicators were just turning on until I blew the fuse. I couldn't find the electronics diagrams for this board, so I was using analagous boards' diagrams and wasn't sure how much current to expect to draw. I swapped out for 1A fuses (only for the electronics I was adding to the system).

2. Now the +24V indicator on the dewhitening board wasn't turning on, and the -24V supply was alternatively drawing ~500mA and 0mA in a ~1Hz square wave. Thinking I could be dropping voltage along the path to the board, I swapped out the cables leading to the whitening/dewhitening boards with 16AWG (was 18AWG). This didn't seem to help.

3. Since the whitening board seemed to be consistently powered on, I removed the dewhitening board to see if there was a problem with it. Indeed, I'd burned out the +24V supply electronics--two resisters were broken entirely, and the breadboard near the voltage regulator had been visibly heated.

  1. I identified that the resistors were 1Ohm, and replaced them (though I couldn't find 1Ohm surface mount resistors). I also replaced the voltage regulator in case it was broken. I couldn't find the exact model, so I replaced the LM2940CT-12 with an LM7812, which I think is the newer 12V regulator.
  2. Though this replacement seemed to work when the power board was disconnected from the dewhitening board, connecting to the dewhitening board again resulted in a lot of current draw.
  3. I depowered the board and decided to take a different approach (see)

I noticed that the +/-15V currents are slightly higher than the labels, but didn't notice whether they were already different before I began this work.

I also noticed one pair of wires in the area of 1X1 I was working that wasn't attached to power (or anything). I didn't know what it was for, so I've attached a picture.

Attachment 1: 52DE723A-02A4-4C62-879B-7B0070AE8A00.jpeg
52DE723A-02A4-4C62-879B-7B0070AE8A00.jpeg
Attachment 2: 545E5512-D003-408B-9F00-55F985966A16.jpeg
545E5512-D003-408B-9F00-55F985966A16.jpeg
Attachment 3: DFF34976-CC49-4E4F-BFD1-A197E2072A32.jpeg
DFF34976-CC49-4E4F-BFD1-A197E2072A32.jpeg
  14339   Mon Dec 10 15:53:16 2018 gautamUpdateLSCSwept-sine measurement with DTT

Disclaimer: This is almost certainly some user error on my part.

I've been trying to get this running for a couple of days, but am struggling to understand some behavior I've been seeing with DTT.

Test:

I wanted to measure some transfer functions in the simulated model I set up.

  • To start with, I put a pendulum (f0 = 1Hz, Q=5) TF into one of the filter modules
  • Isolated it from the other interconnections (by turning off the MEDM ON/OFF switches).
  • Set up a DTT swept-sine measurement
    • EXC channel was C1:OMC-TST_AUX_A_EXC
    • Monitored channels were C1:OMC-TST_AUX_A_IN2 and C1:OMC-TST_AUX_A_OUT.
    • Transfer function being measured was C1:OMC-TST_AUX_A_OUT/C1:OMC-TST_AUX_A_IN2.
    • Coherence between the excitation and output were also monitored.
  • Sweep parameters:
    • Measurement band was 0.1 - 900 Hz
    • Logarithmic, downward.
    • Excitation amplitude = 1ct, waveform = "Sine"

Unexplained behavior:

  • The transfer function measurement fails with a "Synchronization error", at ~15 Hz.
    • I don't know what is special about this frequency, but it fails repeatedly at the same point in the measurement.
  • Coherence is not 1 always
    • Why should the coherence deviate from 1 since everything is simulated? I think numerical noise would manifest when the gain of the filter is small (i.e. high frequencies for the pendulum), but the measurement and coherence seem fine down to a few tens of Hz.

To see if this is just a feature in the simulated model, I tried measuring the "plant" filter in the C1:LSC-PRCL filter bank (which is also just a pendulum TF), and run into the same error. I also tried running the DTT template on donatella (Ubuntu12) and pianosa (SL7), and get the same error, so this must be something I'm doing wrong with the way the measurement is being run / setup. I couldn't find any mention of similar problems in the SimPlant elogs I looked through, does anyone have an idea as to what's going on here?

* I can't get the "import" feature of DTT to work - I go through the GUI prompts to import an ASCII txt file exported from FOTON but nothing selectable shows up in DTT once the import dialog closes (which I presume means that the import was successful). Are we using an outdated version of DTT (GDS-2.15.1)?  But Attachment #1 shows the measured part of the pendulum TF, and is consistent with what is expected until the measurement terminates with a synchronization error.


the import problem is fixed - when importing, you have to give names to the two channels that define the TF you're importing (these can be arbitrary since the ASCII file doesn't have any channel name information). once i did that, the import works. you can see that while the measurement ran, the foton TF matches the DTT measured counterpart.


11 Dec 2pm: After discussing with Jamie and Gabriele, I also tried changing the # of points, start frequency etc, but run into the same error (though admittedly I only tried 4 combinations of these, so not exhaustive).

Attachment 1: SimTF.pdf
SimTF.pdf
  14340   Mon Dec 10 19:47:06 2018 aaronUpdateOMCOMC channels

Taking another look at the datasheet, I don't think LM7812 is an appropriate replacement and I think the LM2940CT-12 is supposed to supply 1A, so it's possible the problem actually is on the power board, not on the dewhitening board. The board takes +/- 15V, not +/- 24...

Quote:
 
  1. I identified that the resistors were 1Ohm, and replaced them (though I couldn't find 1Ohm surface mount resistors). I also replaced the voltage regulator in case it was broken. I couldn't find the exact model, so I replaced the LM2940CT-12 with an LM7812, which I think is the newer 12V regulator.

 

  14341   Tue Dec 11 13:42:44 2018 KojiUpdateOMCOMC channels

FYI:

D050368 Anti-Imaging Chassis
https://dcc.ligo.org/LIGO-D050368

https://labcit.ligo.caltech.edu/~tetzel/files/equip

D050368 Adl SUS/SEI Anti-Image filter board 
S/N 100-102 Assembled by screaming circuits. Begin testing 4/3/06 
S/N xxx Mohana returned it to the shop. No S/N or traveler. Put in shop inventory 4/24/06 
S/N 103 Rev 01. Returned from Screaming circuits 7/10/06. complete except for C28, C29 
S/N 104-106 Rev 01. Returned from Screaming circuits 7/10/06. complete except for C28, C29 Needs DRV-135’s installed 
S/N 107-111 Rev 02 (32768 Hz) Back from assembly 7/14/06 
S/N 112-113 Rev 03 (65536 Hz) assembled into chassis and waiting for test 1/29/07 
S/N 114 Rev 03 (65536 Hz) assembled and ready for test 020507 


D050512 RBS Interface Chassis Power Supply Board (Just an entry. There is no file)

https://dcc.ligo.org/LIGO-D050512

RBS Interface Chassis Power Board D050512-00

https://labcit.ligo.caltech.edu/~rolf/jayfiles/drawings/D050512-00.pdf
 

 

  14342   Tue Dec 11 13:48:04 2018 aaronUpdateOMCOMC channels

Koji gave me some tips on testing this board that I wanted to write down, notes probably a bit intermingled with my thoughts. Thanks Koji, also for the DCC and equipment logging!

  • Test the power and AI boards separately with an external supply, ramping the voltage up slowly for each.
  • If it seems the AI board is actually drawing too much current, may need to check its TPs for where a problem might be
    • If it's really extensive may use an IR camera to see what elements are getting too hot
    • Testing in segments will prevent breaking more components
  • Check the regulator that I've replaced
  • The 1 Ohm resistors may have been acting as extra 1A fuses. i need to make sure the resistors I've used to replace them are rated for >1W, if this is the case.
  • Can check the resistance between +-12V and Gnd inputs on the AI board, if there is a short drawing too much current it may show up there.
  • The 7812 may be an appropriate regulator, but the input voltage may need to be somewhat higher than with the low drop regulator that was used before.
  • I want to double check the diagram on the DCC
  14343   Tue Dec 11 14:24:18 2018 aaronUpdateOMCAligning the OMC

I set up a function generator to drive OMC-L, and have the two DCPD mons and the OMC REFL PD sent to an oscilloscope. I need to select a cds channel over which to read the REFL signal.

The two DCPD mon channels have very different behaviors on the PD mons at the sat box (see attachment). PD1 has an obvious periodicity, PD2 has less noise overall and looks more white. I don't yet understand this, and whether it is caused by real light, something at the PDs, or something at the sat box.

I've again gone through the operations that will happen with the OMC chamber vented. Here's how it'll go, with some of the open questions that I'm discussing with Gautam or whoever is around the 40m:

  1. Function generator is driving OMC-L. Right now there is one 150V Kepco supply in use, located on the ground just to the right of the OMC rack. I only have plans to power it on while scanning OMC-L, and until the OMC is fully in use the standard practice will be to use this HV with two people in the lab and shut it off after the immediate activities.
    1. To do: Is a second drive necessary for the TT drivers? I don't think it is during this vent, because we will want to align into the OMC with the TTs in a 'neutral' state. I recall that the way the TT drivers are set up, 0V from the dac to the driver is the 'centered' position for all TTs. Unless we want to compensate for some known shift of the chambers during pumpdown, I think this is the TT position we should use while aligning the OMMT into the OMC.
    2. To do: make sure I'm driving the right pins with the function generator. Update: Seems I was driving the right channels, here's the pinout.
  2. We will use the reflection of aux from the SRM to align into the OMC.
    1. Gautam pointed out that I hadn't accounted for the recombination BS for the aux beam being 90-10. This means there's actually something like 300uW of aux onto the OMC, rather than ~3mW. This should still be enough to see on a card, so it is fine.
    2. However, the aux beam is aligned to be colinear with the AS beam when the SRM is misaligned. So the question is whether the wedge on the SRM makes the SRM-reflected aux beam not colinear with the AS beam

 

---------

Talked with Gautam for a good while about the above plan. In trying to figure out why the DCPD sat box appears to have a different TF for the two PDs (seems to be some loose cabling problem at the mons, because wiggling the cables changed this), we determined that the AA chassis also wasn't behaving as expected--driving the expected channels (28-31) with a sine wave yields some signal at the 100Hz driving frequency, but all save ch31 were noisy. We also still saw the 100Hz when the chassis was unplugged. I will continue pursuing this, but in the meantime I'm making an IDE40 to DB37 connector so I can drive the ADC channels directly with the DAC channels I've defined (need to match pinouts for D080303 to D080302). I also will make a new SCSI to DB37 adapter that is more robust than mentioned here. I also need to replace the cable carrying HV to the OMC-L driver, so that it doesn't have a wire-to-wire solder joint.

We moved a razor blade on the AP table so it is no longer blocking the aux beam. We checked the alignment of aux into the AS port. AUX and AS are not colinear anywhere on the AP table, and despite confirming that the main AS beam is still being reflected off of the OMC input mirror, the returning AUX beam does not reach the AP table (and probably is not reaching the OMC). AUX needs to be realigned such that it is colinear with the AS beam. It would be good if in this configuration, the SRM is held close to its position when the interferometer is locked, but the TTs should provide us some (~2.5mrad) actuation. Gautam will do this alignment and I will calculate whether the TTs will be able to compensate for any misalignment of the SRM.

Here is the new plan and minimal things to do for the door opening tomorrow:

  1. Function generator is driving OMC-L.
    1. The PZT mon channel is sent to the oscilloscope.
    2. To do: confirm again that the triangle wave I send in results in the expected triangle wave going to the OMC, using this mon channel.
  2. The OMC REFL signal is being sent to the AP PD. See photo.
    1. Need to align into this PD, but this alignment can be done in air on the AP table.
  3. Monitor the DCPD signals using the TPs from the sat box going to the oscilloscope.
    1. There may be further problems with the sat box, but for the initial alignment into the OMC only the REFL signal is necessary.
    2. Not minimally necessary, but the sat box needs a new case. It has a front, back, and bottom, but no main case, so the board is exposed.
  4. I will move the OMMT-to-OMC steering mirrors while watching the scope for flashes in the REFL signal.

That is the first, minimal sequence of steps, which I plan to complete tomorrow. After aligned into the OMC, the alignment into the DCPDs shouldn't need modification. Barring work needed to align from OMC to DCPDs, I think most other work with the OMC can be done in-air.

  14344   Tue Dec 11 14:33:29 2018 gautamUpdateCDSNDScope

NDscope is now running on pianosa. To be really useful, we need the templates, so I've made /users/Templates/NDScope_templates where these will be stored. Perhaps someone can write a parser to convert dataviewer .xml to something ndscope can understand. To get it installed, I had to run:

sudo yum install ndscope
sudo yum install python34-gpstime
sudo yum install python34-dateutil
sudo yum install python34-requests

 I also changed the pythonpath variable to include the python3.4 site-packages library in .bashrc

Quote:

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=44971

Let's install Jamie's new Data Viewer

Attachment 1: ndscope.png
ndscope.png
  14345   Tue Dec 11 18:20:59 2018 gautamUpdateOptical LeversBS/PRM HeNe is dead

I found that the BS/PRM OL SUM channels were reading close to 0. So I went to the optical table, and found that there was no beam from the HeNe. I tried power-cycling the controller, there was no effect. From the trend data, it looks like there was a slow decay over ~400000 seconds (~ 5 days) and then an abrupt shutoff. This is not ideal, because we would have liked to use the Oplevs as a DC alignment reference during the ventnoI plan to use the AS camera to recover some sort of good Michelson alignment, and then if we want to, we can switch out the HeNe.

*How can I export PDF from NDscope?

Attachment 1: BSOL_dead.png
BSOL_dead.png
  14346   Tue Dec 11 22:50:07 2018 aaronUpdateOMCAligning the OMC

I did the following:

  • Noticed that the OMC rack's power has +-18V, but I had tested the HV driver with +-15V. Maybe fine, something to watch.
  • Checked that nothing but the OMC driver board was in use on the OMC's Sorensen (the QPD whitening board in the OMC rack is not in use, and anyway is labeled +-15V), then turned down the rack voltage from 18 to 15V. Photos attached of AUX_OMC_S Sorensen bank.
  • I hadn't used the alternative dither before. I started by driving the alternative dither with a 10Vpp sine wave at 1-10 Hz. I have both the DC and AC driver mons on a scope.
    • Initially, I only give it 10V at the HV. I don't see much, nor at 30V, while driving with 0-10V sine waves between 0.1-100Hz.
    • In my last log, I hadn't been using the alternative dither.
  • Instead, I switch over to the main piezo drive, which is sent over DB9. Now I see the following on the AC/DC piezo mon channels:
    • Increasing the HV input (increasing in steps from 10-50V) yields 1V at the DC piezo mon for 50V at the HV input.
    • Driving under a few 100s of Hz results in no change to the AC dither mon. Driving <1Hz results in a small (~10% for a 10Vpp drive) at the HV. I didn't take a full transfer function, but it is the thing to do with cds.
    • Changing the drive amplitude changes the AC mon amplitude proportionally
    • At a few kHz, the 10Vpp drive saturates the AC mon.
    • Photos are in order:
      • 1Hz drive, visible on the DC mon channel in green
      • 1kHz drive 10Vpp, visible on the AC mon channel in violet
      • 1kHz drive 5Vpp
      • 5kHz drive 10Vpp, saturates the AC mon channel
Attachment 1: 8323029A-970E-4BEA-833E-77E709300446.jpeg
8323029A-970E-4BEA-833E-77E709300446.jpeg
Attachment 2: C52735BB-0C56-41A1-B731-678CDDCEC921.jpeg
C52735BB-0C56-41A1-B731-678CDDCEC921.jpeg
Attachment 3: 3F8A0B3B-5C7C-4876-B3A7-332F560D554D.jpeg
3F8A0B3B-5C7C-4876-B3A7-332F560D554D.jpeg
Attachment 4: 3DC88B6A-4213-4ABD-A890-7EC317D9EED0.jpeg
3DC88B6A-4213-4ABD-A890-7EC317D9EED0.jpeg
Attachment 5: C0C4F9C0-9574-4A17-9414-B99D6E27025F.jpeg
C0C4F9C0-9574-4A17-9414-B99D6E27025F.jpeg
Attachment 6: A191E1DE-552F-42A5-BED7-246001248BBD.jpeg
A191E1DE-552F-42A5-BED7-246001248BBD.jpeg
  14347   Wed Dec 12 11:53:29 2018 aaronUpdateGeneralPower Outage

At 11:13 am there was a ~2-3 second interruption of all power at the 40m.

I checked that nobody was in any of the lab areas at the time of the outage.

I walked along both arms of the 40m and looked for any indicator lights or unusual activity. I took photos of the power supplies that I encountered, attached. I tried to be somewhat complete, but didn't have a list of things in mind to check, so I may have missed something. 

I noticed an electrical buzzing that seemed to emanate from one of the AC adapters on the vacuum rack. I've attached a photo of which one, the buzzing changes when I touch the case of the adapter. I did not modify anything on the vacuum rack. There is also 

Most of the cds channels are still down. I am going through the wiki for procedures on what to log when the power goes off, and will follow the procedures here to get some useful channels.

Attachment 1: IMG_0033.HEIC
Attachment 2: IMG_1027.HEIC
Attachment 3: IMG_2605.HEIC
  14349   Thu Dec 13 01:26:34 2018 gautamUpdateGeneralPower Outage recovery

[koji, gautam]

After several combinations of soft/hard reboots for FB, FEs and expansion chassis, we managed to recover the nominal RTCDS status post power outage. The final reboots were undertaken by the rebootC1LSC.sh script while we went to Hotel Constance. Upon returning, Koji found all the lights to be green. Some remarks:

  1. It seems that we need to first turn on FB
    • Manually start the open-mx and mx services using
      sudo systemctl start open-mx.service 
      sudo systemctl start mx.service
    • Check that the system time returned by gpstime matches the gpstime reported by internet sources.
    • Manually start the daqd processes using
      sudo systemctl start daqd_*
  2. Then fully power cycle (including all front and rear panel power switches/cables) the FEs and the expansion chassis.
    • This seems to be a necessary step for models run on c1sus (as reported by the CDS MEDM screen) to pick up the correct system time (the FE itself seems to pick up the correct time, not sure what's going on here).
    • This was necessary to clear 0x4000 errors.
  3. Power on the expansion chassis.
  4. Power on the FE.
  5. Start the RTCDS models in the usual way
    • For some reason, there is a 1 second mismatch between the gpstime returned on the MEDM screen for a particular CDS model status, and that in the terminal for the host machine.
    • This in itself doesn't seem to cause any timing errors. But see remark about c1sus above in #2.

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14350   Thu Dec 13 10:03:07 2018 ChubUpdateGeneralOMC chamber

Bob, Aaron, and I removed the door from the OMC chamber this morning.  Everything went well.

  14351   Thu Dec 13 12:06:35 2018 gautamUpdateGeneralPower Outage recovery

I did a walkaround and checked the status of all the interlock switches I could find based on the SOP and interlock wiring diagram, but the PSL remains interlocked. I don't want to futz around with AC power lines so I will wait for Koji before debugging further. All the "Danger" signs at the VEA entry points aren't on, suggesting to me that the problem lies pretty far upstream in the wiring, possibly at the AC line input? The Red lights around the PSL enclosure, which are supposed to signal if the enclosure doors are not properly closed, also do not turn on, supporting this hypothesis...

I confirmed that there is nothing wrong with the laser itself - i manually shorted the interlock pins on the rear of the controller and the laser turned on fine, but I am not comfortable operating in this hacky way so I have restored the interlock connections until we decide the next course of action...

Quote:
 

The PSL (Edwin) remains in an interlock-triggered state. We are not sure what is causing this, but the laser cannot be powered on until this is resolved.

  14352   Thu Dec 13 18:12:47 2018 gautamUpdateIOOND filter on AS camera changed

In order to see the AS beam a bit more clearly in our low-power config, I swapped out the ND=1.0 filter on the AS camera for ND=0.5.

  14353   Thu Dec 13 20:10:08 2018 KojiUpdateGeneralPower Outage recovery

[Gautam, Aaron, Koji]

The PSL interlock system was fixed and now the 40m lab is laser hazard as usual.


- The schematic diagram of the interlock system D1200192
- We have opened the interlock box. Immediately we found that the DC switching supply (OMRON S82K-00712) is not functioning anymore.  (Attachment #1)
- We could not remove the module as the power supply was attached on the DIN rail. We decided to leave the broken supply there (it is still AC powered with no DC output).

- Instead, we brought a DC supply adapter from somewhere and chopped the head so that we can hook it up on the crimping-type quick connects. In Attachment #1, the gray is +12V, and the orange and black lines are GND.

- Upon the inspection, the wires of the "door interlock reset button" fell off and the momentary switch (GRAYHILL 30-05-01-502-03) got broken. So it was replaced with another momentary swicth, which is way smaller than the original unfortunately. (Attachments 2 and 3)

- Once the DC supply adapter was pluged to an AC tap, we heard the sounds of the relays working, and we recovered the laser hazard lamps, PSL door alerm lamps. Also it was confirmed that the PSL innolight is operatable now. 

- BTW, there is the big switch box on the wall close to the PSL enclosure. Some of the green lamps were gone. We found that we have plenty of spare lamps and relays inside of the box. So we replaced the bulbs and know the A.C. lights are functioning. (Attachments 4 & 5)

Attachment 1: OMRON_S82K-00712.JPG
OMRON_S82K-00712.JPG
Attachment 2: reset_button_repaired1.JPG
reset_button_repaired1.JPG
Attachment 3: reset_button_repaired2.JPG
reset_button_repaired2.JPG
Attachment 4: gray_box.JPG
gray_box.JPG
Attachment 5: gray_box2.JPG
gray_box2.JPG
  14354   Thu Dec 13 22:24:21 2018 aaronUpdateOMCOMC channels

I completed testing of the AI board mentioned above. In addition to the blown fuse, there were two problems:

  • A was a large drop of solder splattered on some of the ch 1 ICs, which is why we couldn't maintain any voltage. I removed the solder.
  • The +12V wire from the power board to the AI board was loose, so I removed and replaced that crimp connection

After this, I tested the TF of all channels. For the most part, I found the expected 3rd order ~7500Hz cheby with notches at ~16kHz and 32kHz. However, some of the channels had shallower or deeper notches. By ~32kHz, I was below the resolution on the spectrum analyzer. Perhaps I just have nonideal settings? I'll attach a few representative examples.

I reinstalled the chassis at 1X2, but haven't connected power.

 

  14355   Thu Dec 13 22:36:42 2018 aaronUpdateOMCAligning the OMC

I turned on AUX, and aligned the aux beam to be centered on the first optic the AS beam sees on the AP table. I then turned off the AUX laser.

  14356   Thu Dec 13 22:56:28 2018 gautamUpdateCDSFrames

[koji, gautam]

We looked into the /frames situation a bit tonight. Here is a summary:

  1. We have already lost some second trend data since the new FB has been running from ~August 2017.
  2. The minute trend data is still safe from that period, we believe.
  3. The Jetstor has ~2TB of trend data in the /frames/trend folder.
    • This is a combination of "second", "minute_raw" and "minute".
    • It is not clear to us what the distinction is between "minute_raw" and "minute", except that the latter seems to go back farther in time than the former.
    • Even so, the minute trend folder from October 2011 is empty - how did we manage to get the long term trend data?? From the folder volumes, it appears that the oldest available trend data is from ~July 24 2015.

Plan of action:

  1. The wiper script needs to be tweaked a bit to allow more storage for the minute trends (which we presumably want to keep for long term).
  2. We need to clear up some space on FB1 to transfer the old trend data from Jetstor to FB1.
  3. We need to revive the data backup via LDAS. Also summary pages.

BTW - the last chiara (shared drive) backup was October 16 6 am. dmesg showed a bunch of errors, Koji is now running fsck in a tmux session on chiara, let's see if that repairs the errors. We missed the opportunity to swap in the 4TB backup disk, so we will do this at the next opportunity.

  14357   Fri Dec 14 13:02:24 2018 aaronUpdateOMCAligning the OMC

I replaced the 2'' AUX-AS combining BS with a 2'' HR mirror for 1064. I aligned the AUX beam from the new HR mirror into the next iris, so AUX passes through irises both before and after the new optic. Now, AS does not go out to the AS PDs.

I mounted the old BS on the SP table in a random orientation.

I also dumped the beam transmitted through one of the AUX steering mirrors before the new HR mirror.

  14358   Fri Dec 14 13:06:12 2018 aaronUpdateOMCAligning the OMC

I replaced the 2'' AUX-AS combining BS with a freshly mounted 2'' HR mirror for 1064. The mirror is labelled 'Y1-2037-45-P', and had a comment on its case: 'V'. I aligned the AUX beam from the new HR mirror into the next iris, so AUX passes through irises both before and after the new optic. Now, AS does not go out to the AS PDs.

I mounted the old BS on the SP table in a random orientation.

I also dumped the beam transmitted through one of the AUX steering mirrors before the new HR mirror.

  14359   Fri Dec 14 14:25:36 2018 KojiUpdateCDSchiara backup

fsck of chiara backup disk (UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000") was done. But this required many files to be fixed. So the backed-up files are not reliable now.
On the top of that, the disk became not recognized from the machine.

I went to the disk and disconnected the USB and then the power supply, which was/is connected to the UPS.
Then they are reconnected again. This made the disk came back as /media/90a5c98a-22fb-4685-9c17-77ed07a5e000. (*)
After unmounting this disk, I ran "sudo mount -a" to follow the way of mounting as fstab does.
Now I am running the backup script manually so that we can pretend to maintain a snapshot of the day at least.

(*) This is the same situation we found at the recovery from the power shutdown. So my hypothesis is that on Oct 16 at 7 AM during the backup there was a USB failure or disk failure or something which unmounted the disk. This caused some files got damaged. Also this caused the disk mounted as /media/90a5c98a-22fb-4685-9c17-77ed07a5e000. So since then, we did not have the backup.
Update (20:00): The disk connection failed again. I think this disk is no longer reliable.

 

Attachment 1: fsck_log.log
sudo fsck -yV UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000"           [238/276]
[sudo] password for controls:
fsck from util-linux 2.20.1
[/sbin/fsck.ext4 (1) -- /media/40mBackup] fsck.ext4 -y /dev/sde1
e2fsck 1.42 (29-Nov-2011)
/dev/sde1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error reading block 527433852 (Attempt to read block from filesystem resulted in
 short read) while getting next inode from scan.  Ignore error? yes

... 283 more lines ...
  14360   Fri Dec 14 22:19:22 2018 KojiUpdateGeneralChiara new USB 4TB DIsk

Edit: It was not 4TB disk but 6TB disk in fact. (We actually ordered 4TB disk...)

I think the problem of the backup disk was the flaky power supply for the external drive.
I swapped the drive to a new HGST 4TB one, but it was neither recognized nor spun up with the external power supply we had. So I decided to put both the new and old drives in the PC chassis to power them up with the internal power supply. I tested the old disk via a USB-SATA cable. However, this disk was not recognized. I noticed that the disk was not HGST 4TB but Seagate 3TB. Is it possible? I thought it was 4TB... Did I miss something?

Once the new 4TB was connected to the USB-SATA, it was very smooth to get it mounted. Now the disk is mounted as /media/40mBackup as before. /etc/fstab was also modified with the new UUID. All the command logs are found here below.

Let's see how the morning backup goes. It would take a while to copy everything on the new disk. So it was actually very nice to set this disk up by Friday midnight.


controls@chiara|~> lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk 
---- sda1   8:1    0 446.9G  0 part /
---- sda2   8:2    0     1K  0 part 
---- sda5   8:5    0  18.9G  0 part [SWAP]
sdb      8:16   0   1.8T  0 disk 
---- sdb1   8:17   0   1.8T  0 part 
sdc      8:32   0   3.7T  0 disk 
---- sdc1   8:33   0   3.7T  0 part /home/cds
sr0     11:0    1  1024M  0 rom  
sdd      8:64   0   5.5T  0 disk 

controls@chiara|~> sudo mkfs -t ext4 /dev/sdd1

mke2fs 1.42 (29-Nov-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
183144448 inodes, 1465130385 blocks
73256519 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
44713 block groups
32768 blocks per group, 32768 fragments per group
4096 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done      

controls@chiara|~> blkid

/dev/sda1: UUID="972db769-4020-4b74-b943-9b868c26043a" TYPE="ext4" 
/dev/sda5: UUID="a3f5d977-72d7-47c9-a059-38633d16413e" TYPE="swap" 
/dev/sdc1: UUID="92dc7073-bf4d-4c58-8052-63129ff5755b" TYPE="ext4" 
/dev/sdd1: UUID="1843f813-872b-44ff-9a4e-38b77976e8dc" TYPE="ext4" 

controls@chiara|~> sudo emacs -nw /etc/fstab
controls@chiara|~> cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=972db769-4020-4b74-b943-9b868c26043a /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=a3f5d977-72d7-47c9-a059-38633d16413e none            swap    sw              0       0
#UUID="90a5c98a-22fb-4685-9c17-77ed07a5e000"    /media/40mBackup       ext4      defaults,relatime,commit=60       0         0
UUID="1843f813-872b-44ff-9a4e-38b77976e8dc"    /media/40mBackup       ext4      defaults,relatime,commit=60       0         0

#fb:/frames      /frames nfs     ro,bg


UUID=92dc7073-bf4d-4c58-8052-63129ff5755b   /home/cds    ext4    defaults,relatime,commit=60    0   0

controls@chiara|~> sudo mount -a
controls@chiara|~> df

Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda1       461229088   10694700  427105320   3% /
udev             15915020         12   15915008   1% /dev
tmpfs             3185412        868    3184544   1% /run
none                 5120          0       5120   0% /run/lock
none             15927044        484   15926560   1% /run/shm
/dev/sdc1      3845709644 1809568856 1840789912  50% /home/cds
/dev/sdd1      5814346836     190408 5521130352   1% /media/40mBackup
  14361   Sat Dec 15 18:29:53 2018 KojiUpdateGeneralChiara new USB 4TB DIsk

The local backup was done at 18:18 after 11h18m of running.

2018-12-15 07:00:01,699 INFO       Updating backup image of /cvs/cds
2018-12-15 18:17:56,378 INFO       Backup rsync job ran successfully, transferred 5717707 files.

 

  14362   Sat Dec 15 20:04:03 2018 gautamUpdateIOOTT1/TT2 stepping

I'm running a script that moves TT1 and TT2 randomly in some restricted P/Y space to try and find an alignment that gets some light onto the TRY PD. Test started at gpstime 1228967990, should be done in a few hours. The IMC has to remain locked for the duration of this test. I will close the PSL shutter once the test is done. Not sure if the light level transmitted through the ITM, which I estimate to be ~30uW, will be enough to show up on the TRY PD, but worth a shot I figure.

Test was completed and PSL shutter was closed at 1228977122.

  14363   Mon Dec 17 20:45:40 2018 aaronUpdateOMCAligning the OMC

[gautam, aaron]

We did work in the OMC chamber today to get the OMC aligned. Aaron was in the clean suit while Gautam steered in-air optics. We modified the aux input steering optics and the final two OMC steering optics (between OMMT and OMC), but did not modify any of the AS path optics.

I had already aligned AUX approximately into the AS port from the AP table. With the OMC N door open, we aligned the aux beam first to OM6, then to OMPO, then OM5. OM5 was the last optic in the OMC chamber that we could align to.

From there, Gautam found the aux beam clipping on a few optics on its way to SR4 using the IR viewer. Once we were approximately hitting SR4, we got a return beam in the OMC chamber, which we were able to coalign with the input aux beam.

We had already done the alignment of SR5 into the OMC during the last vent, so we immediately had a refl off of the OMC, which we aligned onto a PD520 from the PSL table (larger aperture than the previous PD, which anyway needed a macroscopic adjustment to catch the refl beam).

Next, we removed the OMC cover, wrapped it in foil, and placed it in the makeshift clean room near the Y end. The screws remain in a foil bucket in the OMC chamber. With the cover off, Aaron moved the OMC input steering mirrors to align the beam in the OMC. We measured ~2.4mW in the OMC refl beam, which means about 240uW is transmitted into the OMC. Aaron thinks the beam overlaps itself after one round trip in the cavity, but that the entire plane may be too low in pitch, so more alignment may be needed here.

With the beam approximately aligned into the OMC, we energized the OMC-L piezo driver with 200V, and applied a ~0.03Hz triangle wave on the OMC diff input (pins 2-7). We monitor the REFL PD, piezo mon, function generator signal, and one of the trans PDs. We noticed that the PZT mon shows the driver saturating before the function generator reaches its full +-10V, which is something to investigate.

We saw what could have been regular dips in the REFL PD signal, but realized that with an unkown level of mode matching, it will be hard to tell whether the light becomes resonant with the DC signal. Gautam has suggested coaligning the aux and PSL beams, then observing the PDH signal from the PSL beam as the OMC sweeps through resonance, while turning aux back on anytime we try to make adjustments to the alignment of the OMC (so I can see the beam in the cavity).

I'll think through the plan in some more detail and we will try to have the OMC locked tomorrow.


gautam:

  1. All references to SR4 etc actually refer to OM4 etc.
  2. For this experiment, we are using the prompt reflection of the AUX beam from the HR surface of the SRM, so as to get maximum light into the cavity.
  3. For 2.4 mW incident on OMC1 (actually we measured the REFL beam on the AS table), we expect ~24 uW inside the cavity, which isn't a lot but still was visible on the card.
  4. After this work, I checked the IMC alignment - it was still easily able to lock to a TEM00 mode, but the transmission was ~half (i.e. ~600cts) of what I am used to it being in low power mode (~1300 cts). I didn't align the cavity to the input beam, as I think in this case, the right thing to do is to align the input beam to the IMC cavity axis.
Attachment 1: IMG_5922.JPG
IMG_5922.JPG
Attachment 2: IMG_5921.JPG
IMG_5921.JPG
  14364   Tue Dec 18 11:42:40 2018 ChubUpdateGeneralAcromag box wired

The Auxiliary DAQ Chassis, or Acromag box, is now wired and ready for testing.  I will be sorting the cables at the vacuum rack to make connection to the box easier.

 

  14365   Tue Dec 18 18:13:32 2018 aaronUpdateOMCOMC L HV Piezo driver tests (again)

I tested the OMC-L HV driver box again, and made the following observations:

  • Drove the HV diff pins (2,7) with a 5V triangle wave
    • Observed that with a ~0.4V offset on the drive, the HV output (measured directly with a 10x probe) has a 0-(almost)200V triangle wave (for 200V HV in), saturating near 200V and near 0V somewhat before reaching the full range of the triangle
    • The HV mon gives the same answer as measuring the HV output directly, and is reduced 100x compared to the HV output.
    • At 1Hz and above, the rolloff of the low pass still attenuates the drive a bit, and we don't reach the full range.
  • Drove the HV dither pins (1,6) with a 100mV to 10V triangle wave, around 15kHz
    • Even at 10V, the dithering is near the noise of the mon channel, so while I could see a slight peak changing on the FFT near the dither frequency, I couldn't directly observe this on a scope using the mon channel
    • However, measuring the HV directly I do see the dither applied on top of the HV signal. The amplitude of the dither is the same on the HV output as on the dither drive.

[gautam, aaron]

We searched for blips while nominally scanning the OMC length.

We sent a 0.1Hz, 10Vpp triangle wave to the OMC piezo drive diff channels, so the piezo length is seeing a slow triangle wave from 0-200V.

Then, we applied a ~15kHz dither to the OMC length. This dither is added directly onto the HV signal, so the amplitude of the dither at the OMC is the same as the amplitude of the dither into the HV driver.

We monitored the OMC REFL signal (where we saw no blips yesterday) and mixed this with the 15kHz dither signal to get an error signal. Gautam found a pomona box with a low pass filter, so we also low passsed to get rid of some unidentified high frequency noise we were seeing (possibly a ground loop at the function generator? it was present with the box off, but gone with the AC line unplugged). [So we made our own lock-in amplifier.] Photo attached.

We tested the transfer function of the LP, and finding it at 100kHz rather than the advertised 10kHz, we opened the box, removed a resistor to change the 3dB back to 10kHz, and confirmed this by measuring the TF.

We didn't see flashes of error signal in the mixed reflection either, so we suspect that either the PZT is not actuating on the OMC or the alignment is bad. Based on what appears to be the shimmering of far-misaligned fringes on the AS camera, Aaron's suspicion from aligning the cavity with the card, and the lack of flashes, we suspect the alignment. To avoid being stymied by a malfunctioning PZT, we can scan the laser frequency next time rather than the PZT length.

Attachment 1: IMG_4576_copy.jpg
IMG_4576_copy.jpg
  14366   Wed Dec 19 00:12:46 2018 gautamUpdateOMC40m OMC DCC node

I made a node to collect drawings/schematics for the 40m OMC, added the length drive for now. We should collect other stuff (TT drivers, AA/AI, mechanical drawings etc) there as well for easy reference.

Some numbers FTR:

  • OMC length PZT capacitance was measured to be 209 nF.
  • Series resistance in HV path of OMC lenght PZT driver is 10 kohms, so this creates a LP with corner 1/2/pi/10kohm/200nF ~80 Hz.
  • Per Rob's thesis, the length PZT has DC actuation coefficient of 8.3 nm/V, ∼ 2 µm range. 
  14368   Wed Dec 19 15:15:56 2018 gautamUpdateIOOTT1/TT2 stepping

I removed the ND filter from the ETMYT camera to facilitate searching for a TRY beam. This should be replaced before we go back to high power.

  14369   Wed Dec 19 19:51:19 2018 gautamUpdateGeneralPumpdown prep

[Koji, gautam]

Summary:

We are ready to put the heavy doors back on the chambers and do some test pumpdowns tomorrow morning if Jon gives us the go-ahead. Also, Koji made the OMC resonate some of the AUX beam light we send into ityes

Details:

  1. EY work:
    • IMC was locked, and we attempted to locate the beam with an IR card inside the chamber.
    • Koji found that the beam was too high, we were over-shooting the entire black-glass baffle on the EY table.
    • So I moved the TTs to try and center the beam through the aperture of aforementioned baffle.
    • Once this was done, we found that the beam was misaligned in yaw by ~1-inch in transmission on the EY optics table (there was an iris in place marking the cavity transmission axis). This explains why I couldn't find any TRY flashes while moving the TTs around.
    • We hypothesize that without the 2 degree ETM wedge in place, there isn't a compatible axis for the ITM transmission to also make it through the EY baffle and transmission iris. Over ~1m, the 2 degree wedge makes roughly 1.4 inch translation in yaw, so this seems to be a plausible hypothesis.
    • The ETMY suspension was moved from the mini-cleanroom setup back into the EY vacuum chamber. Two clamps (finger tightened only) hold it in place on the NE edge of the optical table. We decided that this is a better resting palce for the cage over the holidays than an in-air cleanroom.
  2. OMC chamber work:
    • While we were in clean garb, we decided to also investigate the OMC situation a bit.
    • It quickly became apparent that it was hopeless for me to work in chamber in the tightly confined IOO chamber. So Koji went in to have a look.
    • Koji will post the detailed alignment procedure - but after some alignment of the AUX laser input beam axis using in air steering mirrors and Koji's expert tweaking of the pointing into the OMC, we observed some resonances of the OMC.
    • Attachment #1 shows the full-range triangle ramp applied to the OMC length PZT (top row) and the OMC REFL signal (bottom row), measured using a PDA520 (chosen for its large active area) connected to a scope (AC-coupled, 1Mohm impedance, averaged to make the dips more prominent).
    • The OMC transmission was also (barely) visible on an IR card.
    • So the OMC length PZT seems capable of sweeping the length of the cavity. Based on the size of the dips we saw, the MM into the cavity is sub 1-percent.
    • The transmission PDs didn't output any measurable signal - but I'm not sure that the satellite box / readout electronics have been carefully characterized on the electroncis bench, so that will have to be done first.
    • We replaced the copper cover of the OMC (finger tightened for now) in case we do any test pumpdowns tomorrow. HV supply has been turned off, and the AUX laser has been reverted to standby mode.
Attachment 1: OMCscan.pdf
OMCscan.pdf
  14370   Wed Dec 19 21:14:50 2018 gautamUpdateVACPumpdown tomorrow

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

  14371   Wed Dec 19 22:11:28 2018 KojiUpdateGeneralHow to align the copper OMC

The OMC input optics layout is attached

Checked the spot position on OMMT-FM1. It was off from the center. This was causing the spot on OMMT1 off-center. This was fixed by the steering mirror for the AUX laser.

The beam alignment onto the OMC was tweaked with OMC-SM1 and OMC-SM2. This was the painful part. We had to make a sensor card that could get in to the narrow space of the OMC. (Attachment 2 right)

Attachment 2 left shows the naming convention of the OMC mirrors.

For the alignment, we gave 5Vpp trig waves at 3.1Hz to the input of the PZT amp so that the cavity is kept scanned continuously. Firstly check the rough spot positions for OMC-CM1 and OMC-CM2. If you carefully use the card, you can check if the beam is returning to OMC-IC. This return beam should have roughtly same hight as the incident beam. This can be adjusted by either of the steering mirrors.

Once the beam is going around the mirrors multiple times, the spot alignment can be checked at OMC-CM1. Bring a card right in front of CM1. If the card is lifter slightly above the incident spot, this automatically allows for the outgoing beam to go through. Depending on the pitch alignment, the next roundtrip (1RT) will be seen on the card. As you lift the card up more, you will be able to see more round trip beams (e.g. 2RT, 3RT, in the figure). If the yaw alignment is perfect, these spots would be lined up vertically. So you can try to align the horizontal direction with the steering mirrors. Then the vertical alignment can be done with the pitch knobs.

At this point you should be able to see some super high-order transmission at the OMC trans. For today, we stopped here as we already ran out of the knob ranges at multiple knobs. This is because the beam height in the mode matching telescope was not right, and the steering mirrors had to work more than their range.

Attachment 1: 110804_40m_OMC_layout.pdf
110804_40m_OMC_layout.pdf
Attachment 2: OMC_alignment.pdf
OMC_alignment.pdf
  14372   Thu Dec 20 08:38:27 2018 JonUpdateVACPumpdown tomorrow

Linked is the pumpdown procedure, contained in the old 40m documentation. The relevant procedure is "All Off --> Vacuum Normal" on page 11.

Quote:

I just spoke to Jon who asked me to make this elog - we will be ready to test one or more parts of the pumpdown procedure tomorrow (12/20), so we should proceed as planned to put the heavy doors back on EY and OMC chambers at 9am tomorrow morning. Jon will circulate a more detailed procedure about the pumpdown steps later today evening.

 

  14373   Thu Dec 20 10:28:43 2018 gautamUpdateVACHeavy doors back on for pumpdown 82

[Chub, Koji, Gautam]

We replaced the EY and IOO chamber heavy doors by 10:10 am PST. Torquing was done first oen round at 25 ft-lb, next at 45 ft-lb (we trust the calibration on the torque wrench, but how reliable is this? And how important are these numbers in ensuring a smooth pumpdown?). All went smooth. The interior of the IOO chamber was found to be dirty when Koji ran a wipe along some surfaces.

For this pumpdown, we aren't so concerned with having the IFO in an operating state as we will certainly vent it again early next year. So we didn't follow the full close-up checklist.

Jon and Chub and Koji are working on starting the pumpdown now... In order to not have to wear laser safety goggles while we closed doors and pumped down, I turned off all the 1064nm lasers in the lab.

  14374   Thu Dec 20 17:17:41 2018 gautamUpdateCDSLogging of new Vacuum channels

Added the following channels to C0EDCU.ini:

[C1:Vac-P1b_pressure]
units=torr
[C1:Vac-PRP_pressure]
units=torr
[C1:Vac-PTP2_pressure]
units=torr
[C1:Vac-PTP3_pressure]
units=torr
[C1:Vac-TP2_rot]
units=kRPM
[C1:Vac-TP3_rot]
units=kRPM

Also modified the old P1 channel to

[C1:Vac-P1a_pressure]
units=torr

Unfortunately, we realized too late that we don't have these channels in the frames, so we don't have the data from this test pumpdown logged, but we will have future stuff. I say we should also log diagnostics from the pumps, such as temperature, current etc. After making the changes, I restarted the daqd processes.


Things to add to ASA wiki page once the wiki comes back online:

  1. What is the safe way to clean the cryo pump if we want to use it again?
  2. What are safe conditions to turn the RGA on?
  14376   Fri Dec 21 11:11:51 2018 gautamUpdateCDSLogging of new Vacuum channels

The N2 pressure channel name was also wrong in C0EDCU.ini, so I updated it this morning to the correct name and units:

[C1:Vac-N2_pressure]
units=psi

Now it too is being recorded to frames.

  14385   Fri Jan 4 15:18:15 2019 KojiUpdateGeneralChiara disk clean up and internally mounted

[Koji Gautam]

Took the opprtunity of the power glitch to take care of the disk situation of chiara.

- Unmounted /cvs/cds from nodus. This did not affect the services on nodus as they don't use /cvs/cds

- Go to chiara, shut it down, and physically checked the labels of the drives.

root = 0.5TB
/cvs/cds = 4TB HGST
backup of /cvs/cds= 6TB HGST

- These three disks are internally mounted and connected with SATA. Previously, 6TB was on USB.

- There were two other drives (2TB and 3TB) but they seemed logically or physically broken. These two disks were removed from chiara. (they came back online after reformatting on mac. So they seem still physically alive).

controls@chiara|~> df
df: `/var/lib/lightdm/.gvfs': Permission denied
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/sda1       461229088   10690932  427109088   3% /
udev             15915020          4   15915016   1% /dev
tmpfs             3185412        848    3184564   1% /run
none                 5120          0       5120   0% /run/lock
none             15927044        144   15926900   1% /run/shm
/dev/sdb1      5814346836 1783407788 3737912972  33% /media/40mBackup
/dev/sdc1      3845709644 1884187232 1766171536  52% /home/cds
controls@chiara|~> lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk
├─sda1   8:1    0 446.9G  0 part /
├─sda2   8:2    0     1K  0 part
└─sda5   8:5    0  18.9G  0 part [SWAP]
sdb      8:16   0   5.5T  0 disk
└─sdb1   8:17   0   5.5T  0 part /media/40mBackup
sdc      8:32   0   3.7T  0 disk
└─sdc1   8:33   0   3.7T  0 part /home/cds
sr0     11:0    1  1024M  0 rom

- Rebooted the machine and just came back without any error. This time the control room machines were not shutdown, but they just recovered the NFS once chiara got back.

Attachment 1: P_20190104_143336.jpg
P_20190104_143336.jpg
Attachment 2: P_20190104_143357.jpg
P_20190104_143357.jpg
  14386   Fri Jan 4 17:43:24 2019 gautamUpdateCDSTiming issues

[J Hanks (remote), koji, gautam]

Summary:

The problem stems from the way GPS timing signals are handled by the FEs and FB. We effected a partial fix:

  • Now, old frame data is no longer being overwritten
  • For the channels that are indeed being recorded now, the correct time stamp is being applied so they can be found in /frames by looking for the appropriate gpstime.

Details:

  • The usual FE/FB power cycling did not fix the problem.
  • The gps time used by FB and associated RT processes may be found by using  cat /proc/gps (i.e. this is different from the system time found by using date, or gpstime).
  • This was off by 2 years.
  • The way this worked up till now was by adding a fixed offset to this time.
    • This offset can be found as a line saying set symm_gps_offset=31622382 in daqdrc.fw (for example)
    • There were similar lines in daqdrc.rcv and daqdrc.dc - however, they were not all the same offset! We couldn't figure out why.
    • All these files live in /opt/rtcds/caltech/c1/target/daqd/.

Changes effected:

  1. First, we tried changing the offset in the daqdrc.fw file only.
    • Incremented it by 24*60*60*365 = number of seconds in a year with no leap seconds/days.
    • This did not fix the problem.
  2. So J Hanks decided to rebuild the Spectracom driver - these commands may not be comprehensive, but I think I got everything).
    • The relevant file is spectracomGPS.c (made a copy of /usr/src/symmetricom-3.3~rc1, called symmetricom-3.3~rc1-patched, this file is in /usr/src/symmetricom-3.3~rc1-patched/include/drv)
    • Added the following lines:
      /* 2018 had no leap seconds or leap days, so adjust for that */
             pHardware->gpsOffset += 31536000;
    • re-built and installed the modified symmetricom driver.
    • Checked that cat /proc/gps now yields the correct time.
    • Reset the gps time offsets in daqdrc.fw, daqdrc.rcv and daqdrc.dc to 0
    • With these steps, the frames were being written to /frames with the correct timestamp.
  3. Next, we checked the timing on the FEs
    • Basically, J Hanks rebuilt the version of the symmetricom driver that is used by the rtcds models to mimic the changes made for FB.
    • This did the trick for c1lsc and c1ioo - cat /proc/gps now returns the correct time on those FEs.
    • However, c1sus remains problematic (it initially reported a GPS time from 2 years ago, and even after the re-installed driver, is 4 days behind) - he suspects that this is because c1sus is the only FE with a Symmetricom/Spectracom card installed in the I/O chassis. So c1sus reports a gpstime that is ~4 days behind the "correct" gpstime.
    • He is going to check with Rolf/other CDS experts to figure out if it's okay for us to simply remove the card and run the models, or if we need to take other steps.
    • As part of this work, the c1x02 IOP model was recompiled, re-installed and re-started.

The realtime models were not restarted (although all the vertex FEs are running) - we can regroup next week and decide what is the correct course of action.

Quote:
 
  • Attachment #2 shows the minute trend of the pressure gauges for a 12 day period - it looks like there is some issue with the frame builder clock, perhaps this issue resurfaced? But checking the system time on FB doesn't suggest anything is wrong.. I double checked with dataviewer as well that the trends don't exist... But checking the status of the individual daqd processes indeed showed that the dates were off by 1 year, so I just restarted all of them and now the time seems correct. How can we fix this problem more permanently? Also, the P1b readout looks suspicious - why are there periods where it seems like we are reading values better than the LSB of the device?
  14389   Tue Jan 8 10:27:27 2019 gautamUpdateGeneralNear-term in-chamber work

Here is a list of tasks I think we should prioritize for the next two weeks. The idea is to get back to the previous state of being able to do single arm, PRMI-on-carrier and DRMI locking, before making further changes.

Once the new folding mirrors arrive, I'd like to modify the SRC length to allow locking in the signal-recycled config as opposed to RSE. Still need to do the detailed layout, but I think the in-vacuum layout will work. In that case, I'd like to also move the OMC and OMMT to the IY table, and also move the in-air AS photodiodes to the IY in-air optical table. This is why I've omitted the OMC alignment from this near-term list, but if we want to not move the OMC, then we probably should add alignment of the AS beam to the OMC to this list.

List of in-chamber tasks for 1/2019

Chamber Task(s)
EY
  • Clean ETMY optic and suspension
  • Put ETMY suspension back in place, recover Y-arm cavity alignment
  • Remove any residual hardware from unused heater setup
  • Restore parabolic heater setup, center radiation pattern as best as possible on ETMY
  • Check beam position on IPANG steering mirror
IY
  • Clean ITMY optic and suspension cage
  • Restore ITMY suspension, recover Y arm cavity alignment.
  • Check position of AS beam on OM1/OM2
BS/PRM (if we decide to open it)
  • Replace BS/PRM Oplev HeNe, bring the beam in and out of vacuum with beam well centerd on in-vacuum mirrors (can take this opportunity to fix the in-air layout as well to minimize un-necessary steering mirrors)
  • Check position of AS beam on OM3/OM4, adjust if necessary
  • Check position of IPPOS and IPANG beams on their respective steering optics
OMC (if we decide to open it)
  • Check position of AS beam on OM5/OM6
  • Ensure AS beam exits the vacuum cleanly
  14390   Tue Jan 8 19:13:39 2019 JonUpdateUpgradeReady for pumpdown tomorrow

Everything is set for a second pumpdown tomorrow. We'll plan to start pumping after the 1pm meeting. Since the main volume is already at 12 torr, the roughing phase won't take nearly as long this time.

I've added new channels for the TP1 analog readings (current and speed) and for the two N2 tank pressure readings. Chub finished installing the new regulator and has run the transducer signal cable to the vacuum rack. In the morning he will terminate the cable and make the final connection to the Acromag.

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged. We also set up a git repo containing all of the Python interlock/interfacing code: https://git.ligo.org/40m/vacpython. The idea is to use the issue tracker to systematically document any changes to the interlock code.

  14391   Wed Jan 9 11:07:09 2019 gautamUpdateVACNew Vac channel logging

Looks like I didn't restart all the daqd processes last night, so the data was not in fact being recorded to frames. I just restarted everything, and looks like the data for the last 3 minutes are being recorded yes. Is it reasonable that the TP1 current channel is reporting 0.75A of current draw now, when the pump is off? Also the temperature readback of TP3 seems a lot jumpier than that of TP2, probably has to do with the old controller having fewer ADC bits or something, but perhaps the SMOO needs to be adjusted.

Quote:
 

Gautam and I updated the framebuilder config file, adding the newly-added channels to the list of those to be logged.

Attachment 1: Screenshot_from_2019-01-09_11-08-28.png
Screenshot_from_2019-01-09_11-08-28.png
  14392   Wed Jan 9 11:33:35 2019 gautamUpdateCDSTiming issues still persist

Summary:

The gps time mismatch between /proc/gps and gpstime seems to be resolved. However, the 0x4000 DC errors still persist. It is not clear to me why.

Details:

On the phone with J Hanks on Friday, he reminded me that c1sus seems to be the only machine with an IRIG-B timing card installed. I can't find the elog but I remembered that Jamie, ericq and I had done this work in 2016 (?), and I also remembered Jamie saying it wasn't working exactly as expected. Since the DAQ was working fine before this card was installed, and since there are no problems with the recording of channels from the other four FE machines without this card installed, I decided to simply pull out the card from the expansion chassis. The card has been stored in the CDS/FE cabinet along the Y arm for now. There was also a cable that interfaces to the card which brings over the 1pps from the GPS unit, which has also been stored in the CDS/FE cabinet.

This seems to have resolved the mismatch between the gpstime reported by cat /proc/gps and the gpstime commands - Attachment #1 (the <1 second mismatch is presumably due to the deadtime between commands). However, the 0x4000 DC errors still persist. I'll try the full power cycle of FEs and FB which has fixed this kind of error in the past, but apart from that, I'm out of ideas.

Update 1215:

Following the instructions in this elog did not fix the problem. The problem seems to be with the daqd_fw service, which reports the following:

controls@fb1:~ 0$ sudo systemctl status daqd_fw.service 
● daqd_fw.service - Advanced LIGO RTS daqd frame writer
   Loaded: loaded (/etc/systemd/system/daqd_fw.service; enabled)
   Active: failed (Result: start-limit) since Wed 2019-01-09 12:17:12 PST; 2min 0s ago
  Process: 2120 ExecStart=/usr/bin/daqd_fw -c /opt/rtcds/caltech/c1/target/daqd/daqdrc.fw (code=killed, signal=ABRT)
 Main PID: 2120 (code=killed, signal=ABRT)

Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service holdoff time over, scheduling restart.
Jan 09 12:17:12 fb1 systemd[1]: Stopping Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: Starting Advanced LIGO RTS daqd frame writer...
Jan 09 12:17:12 fb1 systemd[1]: daqd_fw.service start request repeated too quickly, refusing to start.
Jan 09 12:17:12 fb1 systemd[1]: Failed to start Advanced LIGO RTS daqd frame writer.
Jan 09 12:17:12 fb1 systemd[1]: Unit daqd_fw.service entered failed state.
                                                     

Update 1530:

The frame-writer error was tracked down to a C0EDCU issue. Jon told me that the Hornet CC1 pressure gauge channel was renamed to . C1:Vac-CC1_pressure, and I made the change in the C0EDCU file. However, it returns a value of 9990000000.0, which the frame writer is not happy about... Keeping the old channel name makes the frame-writer run again (although the actual data is bunk).

Update 1755:

J Hanks suggested adding a 1 second offset to the daqdrc config files. This has now fixed the 0x4000 errors, and we are back to the "nominal" RTCDS status screen now - Attachment #2.

Attachment 1: gpstimeSync.png
gpstimeSync.png
Attachment 2: Screenshot_from_2019-01-09_17-56-58.png
Screenshot_from_2019-01-09_17-56-58.png
  14393   Wed Jan 9 20:01:25 2019 JonUpdateVACSecond pumpdown completed

[Jon, Koji, Chub, Gautam]

Summary

The second pumpdown with the new vacuum system was completed successfully today. A time history is attached below.

We started with the main volume still at 12 torr from the Dec. pumpdown. Roughing from 12 to 0.5 torr took approximately two hours, at which point we valved out RP1 and RP3 and valved in TP1 backed by TP2 and TP3. We additionally used the AUX dry pump connected to the backing lines of TP2 and TP3, which we found to boost the overall pump rate by a factor of ~3. The manual hand-crank valve directly in front of TP1 was used to throttle the pump rate, to avoid tripping software interlocks. If the crank valve is opened too quickly, the pressure differential between the main volume (TP1 intake) and TP1 exhaust becomes >1 torr, tripping the V1 valve-close interlock. Once the main volume pressure reached 1e-2 torr, the crank valve could be opened fully.

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

New Vac Control Station

We installed a local controls terminal for the vacuum system on the desk in front of the vacuum rack (pictured below). This console is connected directly to c1vac and can be used to monitor/control the system even during a network outage or power failure. The entire pumpdown was run from this station today.

To open a controls MEDM screen, open a terminal and execute the alias

$control

Similarly, to open a monitor-only MEDM screen, execute the alias

$monitor
Attachment 1: Screenshot_from_2019-01-09_20-00-39.png
Screenshot_from_2019-01-09_20-00-39.png
Attachment 2: IMG_3088.jpg
IMG_3088.jpg
  14394   Thu Jan 10 10:23:46 2019 gautamUpdateVACovernight leak rate

Overnight, the pressure increased from 247 uTorr to 264 uTorr over a period of 30000 seconds. Assuming an IFO volume of 33,000 liters, this corresponds to an average leak rate of ~20 uTorr L / s. It'd be interesting to see how this compares with the spec'd leak rates of the Viton O-ring seals and valves/ outgassing rates. The two channels in the screenshot are monitoring the same pressure from the same sensor, top pane is a digital readout while the bottom is a calibrated analog readout that is subsequently digitized into the CDS system.

Quote:
 

We allowed the pumpdown to continue until reaching 9e-4 torr in the main volume. At this point we valved off the main volume, valved off TP2 and TP3, and then shut down all turbo pumps/dry pumps. We will continue pumping tomorrow under the supervision of an operator. If the system continues to perform problem-free, we will likely leave the turbos pumping on the main volume and annuli after tomorrow.

Attachment 1: OvernightLeak.png
OvernightLeak.png
  14395   Thu Jan 10 11:32:40 2019 ChubUpdateVACManual valve interfaced with CDS

Connected the manual gate valve status indicator to the Acromag box this morning.  Labeled the temporary cable (a 50' 9p DSUB, will order a proper sized cable shortly) and the panel RV2.  

  14396   Thu Jan 10 19:59:08 2019 JonUpdateVACVac System Running Normally on Turbo Pumps

[Jon, Gautam, Chub]

Summary

We continued the pumpdown of the IFO today. The main volume pressure has reached 1.9e-5 torr and is continuing to fall. The system has performed without issue all day, so we'll leave the turbos continuously running from here on in the normal pumping configuration. Both TP2 and TP3 are currently backing for TP1. Once the main volume reaches operating pressure, we can transition TP3 to pump the annuli. They have already been roughed to ~0.1 torr. At that point the speed of all three turbo pumps can also be reduced. I've finished final edits/cleanup of the interlock code and MEDM screens.

Python Code

All the python code running on c1vac is archived to the git repo: 

https://git.ligo.org/40m/vacpython

This includes both the interlock code and the serial device clients for interfacing with gauges and pumps.

MEDM Monitor/Control

We're still using the same base MEDM monitor/control screens, but they have been much improved. Improvements:

  • Valves now light up in red when they are open. This makes it much easier to see at a glance what is valved in/out.
  • Every pump in the system (except CP1) is now digitally controlled from the MEDM control screen. No more need to physically push any buttons in the vaccum rack. 👍
  • The turbo pumps now show additional diagnostic readouts: speed (TP1/2/3), temperature (TP2/3), current draw (TP1/2/3), and voltage (TP2/3).
  • The foreline pressure gauge readouts for TP2/3 have been added to the digital system.
  • The two new main volume gauges, Hornet and SuperBee, have been added to the digital system as well.
  • New transducers have been added to read back the two N2 tank pressures.
  • The interlock code generates a log file of all its actions. A field in the MEDM screens specifies the location of the log file.
  • A tripped interlock (appearing as a message in the "Error message" field) must be manually cleared via the "Clear error message" button on the control screen before the system will accept any more manual valve input.

Note: The apparent glitches in the pressure and TP diagnostic channels are due to the interlock system being taken down to implement some of these changes.

Attachment 1: Screen_Shot_2019-01-10_at_7.58.24_PM.png
Screen_Shot_2019-01-10_at_7.58.24_PM.png
Attachment 2: CCs.png
CCs.png
Attachment 3: TPs.png
TPs.png
  14397   Fri Jan 11 16:38:57 2019 gautamUpdateGeneralSome alignment checks

The pumpdown seems to be progressing smoothly, so I think we are going to stick with the plan decided on Wednesday, and vent the IFO on Monday at 8am. I decided to do some checks of the IFO alignment.

I turned on the PSL again (so goggles are advisable again inside the VEA until this work is done), re-locked the PMC, and opened the PSL shutter into the vacuum (still low power 100 mW beam going into vacuum). The IMC alignment required minor tweaking, but I recovered ~1300 cts transmission which is what it was --> so we didn't macroscopically change the input pointing into the IMC while working on the IOO table.

Centering the ITMY oplev spot, there is a spot on the AS camera roughly centered on the control room monitor, so the TT pointing must also be pretty close.

Then I centered the ITMY oplev spot to check how well-aligned or otherwise the Michelson was - the BS has no Oplev so there was considerable angular motion of the Michelson spot, but it looked like on average, it was swinging around through a well aligned place. I saved the slow bias voltages for the ITMs and BS in this config.

Then I re-aligned ETMX and checked the green transmission - it was okay, ~0.3, and I was able to increase it to ~0.4 using the EX green PZT mirrors. So far so good.

Finally, I tried to lock the X-arm on IR - after zeroing the offsets on the transmission QPD, there seems to be a few flashes as the cavity swings through resonances, but no discernible PDH error signal. Moreover the input pointing of the IR into the X arm is controlled by the BS which is swinging around all over the place right now, so perhaps locking is hopeless, but the overall alignment of the IFO seems not too bad. Once ETMY is cleaned and put back in place, perhaps the Y arm can be locked.

I shuttered the PSL and inserted a manual beam block, and also turned off the EX laser so that we can vent on Monday without laser goggles.

*Not directly related to this work: we still have to implement the vacuum interlock condition that closes the PSL shutter in the event of a vacuum failure. It's probably fine now while the PSL power is attenuated, but once we have the high power beam going in, it'd be a good to revert to the old standard.

Attachment 1: pd82.png
pd82.png
Attachment 2: LSC_X.png
LSC_X.png
  14398   Mon Jan 14 10:06:53 2019 gautamUpdateVACVent 82 complete

[chub, gautam]

  • IFO pressure was ~2e-4 torr when we started, on account of the interlock code closing all valves because the N2 line pressure dropped below threshold (<65 psi)
  • Chub fixed the problem on the regulator in the drill-press area where the N2 tanks are, the N2 line is now at ~75 psi so that we have the ability to actuate valves if we so desire
  • We decided that there is no need to vent the pumpspool this time - avoiding an unnecessary turbo landing, so the pumpspool is completely valved off from the main volume and TPs 1-3 are left running
  • Went through the pre-vent checklist:
    • Chub measured particle count, deemed it to be okay (I think we should re-locate the particle counter to near 1X8 because that is where the air enters the IFO anyways, and that way, we can hook it up to the serial device server and have a computerized record of this number as we had in the past, instead of writing it down in a notebook)
    • Checked that the PSL was manually blocked from entering the IFO
    • Walked through the lab, visually inspected Jam Nuts and window covers, all was deemed okay
  • Moved 2 tanks of N2 into the lab on account of the rain
  • Started the vent at ~930am PST
    • There were a couple of short bursty increases in the pressure as we figured out the right valve settings but on average, things are rising at approx the same rate as we had in vent 81...
    • There was a rattling noise coming from the drypump that is the forepump for TP2 (Agilent) - turned out to be the plastic shell/casing on the drypump, moreover, the TP2 diagnostics (temperature, current etc) are all normal.
    • The CC1 gauge (Hornet) is supposed to have an auto-shutoff of its High Voltage when the pressure exceeds 10 mTorr, but it was reporting pressures in the 1 mTorr range even when the adjacent Pirani was at 25 torr. To avoid risk of damage, we manually turned the HV off. There needs to be a python script that can be executed to transition control between the remote and local control modes for the hornet, we had to Power Cycle the gauge because it wouldn't give us local control over the HV.
    • Transitioned from N2 to dry air at P1a=25 torr. We had some trouble finding the correct regulator (left-handed thread) for the dry air cylinders, it was stored in a cabinet labelled green optics no
    • Disconnected dry air from VV1 intake once P1b reached 700 torr, to let lab air flow into the IFO and avoid overpressuring.
    • VA* and VAV* valves were opened so as to vent the annuli as we anticipate multiple chamber openings for this vent.

As of 8pm local time, the IFO seems to have equilibriated to atmospheric pressure (I don't hear the hiss of in-rushing air near 1X8 and P1a reports 760 torr). The pumpspool looks healthy and there are no signs in the TP diagnostics channels that anything bad happened to the pumps. Chub is working on getting the N2 setup more robust, we plan to take the EY door off at 9am tomorrow morning with Bob's help.

* I took this opportunity to follow instructions on pg 29 of the manual and set the calibration for the SuperBee pirani gauge to 760 torr so that it is in better agreement with our existing P1a Pirani gauge. The correction was ~8% (820-->760).

Attachment 1: Vent82Summary.png
Vent82Summary.png
  14399   Tue Jan 15 10:52:38 2019 gautamUpdateSUSEY door opened

[chub, bob, gautam]

We took the heavy door off the EY chamber at ~930am.

Chamber work:

  • ETMY suspension cage was returned to its nominal position.
  • Unused hardware from the annular heater setup was removed.
  • The unused heater had its leads snipped close to the heater crimp point, and the exposed part of the bare wires was covered with Kapton tape (we should remove the source leads as well in air to avoid any accidental shorting)

Waiting for the table to level off now. Plan for later today / tomorrow is as follows:

  1. Lock the Y arm, recover good cavity alignment.
  2. Position parabolic heater such that clipping issue is resolved.
  3. Move optic to edge of table for FC cleaning
  4. Clean optic
  5. Return suspension cage to nominal position.
ELOG V3.1.3-